Augmented reality for vehicle operations

ABSTRACT

Systems, methods, and computer products according to the principles of the present inventions may involve a training system for a pilot of an aircraft. The training system may include an aircraft sensor system affixed to the aircraft adapted to provide a location of the aircraft, including an altitude of the aircraft, speed of the aircraft, and directional attitude of the aircraft. It may further include a helmet position sensor system adapted to determine a location of a helmet within a cockpit of the aircraft and a viewing direction of a pilot wearing the helmet. The helmet may include a see-through computer display through which the pilot sees an environment outside of the aircraft with computer content overlaying the environment to create an augmented reality view of the environment for the pilot. A computer content presentation system may be adapted to present computer content to the see-through computer display at a virtual marker, generated by the computer content presentation system, representing a geospatial position of a training asset moving within a visual range of the pilot, such that the pilot sees the computer content from a perspective consistent with the aircraft&#39;s position, altitude, attitude, and the pilot&#39;s helmet position when the pilot&#39;s viewing direction is aligned with the virtual marker.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.17/085,809, filed Oct. 30, 2020 and is a continuation-in-part of U.S.application Ser. No. 16/281,513, filed Feb. 21, 2019, is acontinuation-in-part of U.S. application Ser. No. 16/281,499, filed Feb.21, 2019, which claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/690,363, filed Jun. 27, 2018, and of U.S.Provisional Patent Application No. 62/663,883, filed Apr. 27, 2018, andis a continuation-in-part of U.S. application Ser. No. 16/243,026, filedJan. 8, 2019, which claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/663,883, filed Apr. 27, 2018; the entiredisclosures of each of which are hereby incorporated herein byreference.

FIELD OF THE INVENTION

The present disclosure relates generally to the field of augmentedreality. More specifically, the present disclosure describes methods andsystems using augmented reality in fast moving environments.

BACKGROUND OF THE INVENTION

Augmented reality systems generally use inside-out or outside-intracking systems. These systems generally use objects in the environmentas markers to assist in the tracking and to spatially locate content.There are environments where using such markers is difficult or notpossible. Therefore, there is a need for improved augmented realitysystems and methods.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form, that are further described below in the DetailedDescription. This summary is not intended to limit the claimed subjectmatter's scope.

In accordance with an exemplary and non-limiting embodiment, a methodcomprises presenting a generated first virtual environment to a pilotoperating an aircraft, detecting and saving data indicative of one ormore reactions by the pilot in response to the first virtualenvironment, analyzing the saved data to determine a correlation betweenan identified attribute of the virtual environment and at least one ofthe one or more reactions and presenting a second generated virtualenvironment comprising the identified attribute to the pilot to inducethe at least one of the one or more reactions by the pilot.

Both the foregoing summary and the following detailed descriptionprovide examples and are explanatory only. Accordingly, the foregoingsummary and the following detailed description should not be consideredto be restrictive. Further, features or variations may be provided inaddition to those set forth herein. For example, embodiments may bedirected to various feature combinations and sub-combinations describedin the detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various embodiments of the presentdisclosure. The drawings may contain text or captions that may explaincertain embodiments of the present disclosure. This text is included forillustrative, non-limiting, explanatory purposes of certain embodimentsdetailed in the present disclosure.

FIG. 1 is an illustration of an online platform in accordance withembodiments of the present disclosure.

FIG. 2 illustrates a system to allow real pilots in real aircraft usingaugmented reality to meet in a virtual piece of airspace, in accordancewith embodiments of the present disclosure.

FIG. 3 is a block diagram of a system for facilitating provisioning of avirtual experience, in accordance with embodiments of the presentdisclosure.

FIG. 4 is a block diagram of a first head mount display for facilitatingprovisioning of a virtual experience, in accordance with embodiments ofthe present disclosure.

FIG. 5 is a block diagram of an apparatus for facilitating provisioningof a virtual experience, in accordance with embodiments of the presentdisclosure.

FIG. 6 is a flowchart of a method of facilitating provisioning of avirtual experience, in accordance with embodiments of the presentdisclosure.

FIG. 7 shows a system for facilitating provisioning of a virtualexperience, in accordance with embodiments of the present disclosure.

FIG. 8 is a flowchart of a method of facilitating provisioning of avirtual experience, in accordance with embodiments of the presentdisclosure.

FIG. 9 is a flowchart of a method to facilitate providing a firstpresentation data, in accordance with embodiments of the presentdisclosure.

FIG. 10 illustrates a method to allow real pilots in real aircraft usingaugmented and virtual reality to meet in a virtual piece of airspace, inaccordance with embodiments of the present disclosure.

FIG. 11 shows an augmented reality view shown to a real pilot, inaccordance with embodiments of the present disclosure.

FIG. 12 shows two real aircraft in a virtual airspace, in accordancewith embodiments of the present disclosure.

FIG. 13 shows an augmented reality view shown to a real pilot, inaccordance with embodiments of the present disclosure.

FIG. 14 is a chart related to the United States airspace system'sclassification scheme.

FIG. 15 shows an augmented reality view shown to a real pilot whileaircraft is taxiing at an airport, in accordance with embodiments of thepresent disclosure.

FIG. 16 is a block diagram of a computing device for implementing themethods disclosed herein, in accordance with embodiments of the presentdisclosure.

DETAIL DESCRIPTIONS OF THE INVENTION

While systems and methods are described herein in detail in relation toone or more embodiments, it is to be understood that this disclosure isillustrative and exemplary, and are made merely for the purposes ofproviding a written and enabling disclosure. The detailed disclosureherein is not intended, nor is to be construed, to limit the scope ofpatent protection afforded in any claim of a patent issuing here from,which scope is to be defined by the claims and the equivalents thereof.

The following detailed description refers to the accompanying drawings,which are incorporated herein.

Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar elements.While specific embodiments of the disclosure may be described in detail,modifications, adaptations, and other implementations are foreseeable bythe inventors. For example, substitutions, additions, or modificationsmay be made to the elements illustrated in the drawings, and the systemsand methods described herein may be modified by substituting,reordering, or adding stages to the disclosed methods. Accordingly, thefollowing detailed description does not limit the disclosure. Instead,the proper scope of the disclosure is defined by the appended claims.The present disclosure may contain headers. It should be understood thatthese headers are used as references and are not to be construed aslimiting upon the subjected matter disclosed under the header.

The present disclosure includes many aspects and features. Moreover,while many aspects and features relate to, and are described in thecontext of augmented and/or virtual reality, embodiments of the presentdisclosure are not limited to use only in this context.

Location, Prediction, Presentation

The inventors discovered that augmented reality systems are not capableof locking geospatially located augmented reality content in a positionwithin an environment that are absent real objects or has limitedobjects. Imagine that you are flying a plane at 10,000 feet above theground. The pilot's view is wonderful, but it may absent any realobjects that are geolocated with any precision. The pilot may seeclouds, the sun, other planes temporarily, but the pilot does not seeobjects that are generally used to anchor content, such as walls,outdoor geolocated buildings, mapped roads, etc. The inventors furtherdiscovered that in such environments the systems, in embodiments,required precision location of the user, precision identification ofwhere the user is looking and tracking of these attributes in real-timesuch that the geolocated content can be more precisely fixed inposition. Add to this problem, as the inventor's discovered, that whenpresenting augmented reality content to a fast-moving vehicle in such anenvironment the issues get even more challenging. Systems and methodsdiscovered by the inventors may be used in such environments or even inenvironments where there are real objects that could be used foranchoring of virtual content. Systems and methods in accordance with theprinciples of the present inventions may relate to a situation referredto as ‘within visual range’ of a vehicle. Training within visual rangeis generally training based on up to approximately 10 miles from anaircraft because that is approximately how far a pilot can see on aclear day. The training may involve presenting visual information in theform of augmented reality content to the pilot where the augmentedreality content represents a training asset within the pilot's visualrange.

Embodiments of the present invention may provide systems and methods fortraining of a pilot in a real aircraft while flying and performingmaneuvers. Such a system may include an aircraft sensor system affixedto the aircraft adapted to provide a location of the aircraft, includingan altitude of the aircraft, speed of the aircraft, and directionalattitude of the aircraft, etc. The system may also include a headmounted display (HMD) sensor system (e.g. helmet position sensor system)adapted to determine a location of HMD within a cockpit of the aircraftand a viewing direction of a pilot wearing the helmet. The HMD may havea see-through computer display through which the pilot sees anenvironment outside of the aircraft with computer content overlaying theenvironment to create an augmented reality view of the environment forthe pilot. The system may include a computer content presentation systemadapted to present computer content to the see-through computer displayat a virtual marker, generated by the computer content presentationsystem, representing a geospatial position of a training asset movingwithin a visual range of the pilot, such that the pilot sees thecomputer content from a perspective consistent with the aircraft'sposition, altitude, attitude, and the pilot's helmet position when thepilot's viewing direction is aligned with the virtual marker. Thevirtual marker may represent one in a series of geospatial locationsthat define the movement of the training asset and one of the series maybe used as an anchor for the presentation of the virtual training assetcontent in a frame at a time representing a then current time.

In embodiments, the computer content represents a virtual asset in atraining exercise for the pilot. The pilot may use the aircraft controlsto navigate the aircraft in response to the virtual asset's location ormovement. The computer content presentation system may receiveinformation relating to the pilot's navigation of the aircraft andcauses the virtual asset to react to the navigation of the aircraft. Thereaction may be selected from a set of possible reactions and/or basedon artificial intelligence systems. The virtual training asset may be avirtual aircraft, missile, enemy asset, friendly asset, ground asset,etc.

In embodiments, the augmented reality content's virtual marker'sgeospatial position is not associated with a real object in theenvironment. The environment may or may not have real objects in it, butthe virtual marker may not be associated with the real object. Theinventor's discovered that augmented reality content is generally lockedinto a location by using a physical object in the environment as ananchor for the content. For example, generally the content may beassociated or ‘connected’ with a building, wall, street, sign, or otherobject that is either mapped to a location or not. A system or methodaccording to the principles of the present invention may lock thecontent to a virtual marker in the air such that it can represent avirtual object can be presented as being in the air without beingassociated with an object in the environment. The apparent stability ofsuch content, as viewed from an operator of a vehicle, may depend onmaintaining an accurate geometric understanding of the relative positionof the operator's HMD and the content virtual marker's geospatiallocation. A main cause of error in maintaining the geometricunderstanding may be maintaining an accurate understanding of thevehicle's position, attitude, speed, vibrations, etc. The geometricunderstanding between the vehicle and the geospatially located virtualmarker may be accurate if the vehicle's location and condition is wellunderstood. In embodiments, the geometric understanding changes quicklybecause both the vehicle and the virtual marker may be moving throughthe environment. For example, the vehicle may be a jet fighter aircraftmoving at 800 miles per hour and the augmented reality content mayrepresent an antiaircraft missile moving at 1500 miles an hour towardsthe aircraft. In such a training simulation both the real aircraft andvirtual content are moving very fast and the relative geometry betweenthem is changing even faster. A system and method according to theprinciples of the present invention update the relative geometricunderstanding describing the relationship between the vehicle and thevirtual marker. The system may further include in the relative geometricunderstanding the vehicle operator's head location and viewing positionand/or eye position. To maintain an accurate geometric understanding, asystem and method may track information from sensors mounted within thevehicle, including a one or more sensors such as GPS, airspeed sensor,vertical airspeed sensor, stall sensor, IMU, G-Force sensor, avionicssensors, compass, altimeter, angle sensor, attitude heading andreference system sensors, angle of attack sensor, roll sensor, pitchsensor, yaw sensor, force sensors, vibration sensors, gyroscopes, enginesensors, tachometer, control surface sensors, etc.

Systems and methods according to the principles of the presentinventions may include a helmet position sensor system that includes aplurality of transceivers affixed within the aircraft adapted totriangulate the location and viewing direction of the helmet. Theplurality of transceivers may operate at an electromagnetic frequencyoutside the visible range. The helmet may include at least one markeradapted to be recognized by the triangulation system for theidentification the helmet location and helmet viewing direction. Forexample, the helmet may have several markers on it at known positionsand three or more electromagnetic transceivers may be mounted at knownlocations in the cockpit of an aircraft, or operator's environment in avehicle. The transceivers each measure, through time of flightmeasurements, the distance between each transceiver and the marker(s) onthe helmet and then the measurements may be used to triangulate thelocation and viewing position of the helmet. In embodiments, the helmetmay be markerless and the triangulation system may ‘image’ the helmet tounderstand it's location and position.

Systems and methods according to the principles of the presentinventions may include a helmet position sensor system that triangulatesthe helmet position by measuring a plurality of distances from thehelmet (or other HMD) to known locations within the aircraft. This maygenerally be referred to as an inside out measurement. The knownlocations may include a material with a particular reflectioncharacteristic that is matched with the transceiver system in thehelmet.

As disclosed herein, the augmented reality content presented to anoperator of a vehicle may be presented based on the physical environmentthat the vehicle is actually in or it may be based on a differentenvironment such as an environment of another aircraft involved in thesimulated training but is geographically remote from the operator. Insuch a situation, the virtual content presented to the operator may beinfluenced by the other vehicle's environment. For example, a firstaircraft may be flying in a cloudy environment and a second aircraft maybe flying in a bright sunny sky. The first aircraft may be presented avirtual environment based on the second aircraft's actual environment.While the pilot of the second aircraft may have to deal with the brightsun at times, the pilot of the first may not. The virtual contentpresentation system may present the same virtual training asset to boththe first and second pilots, but the content may be faded to mimic adifficult to see asset due to the sun. The computer content may have abrightness and contrast, and at least one of the brightness and contrastmay be determined by the pilot's viewing direction when the content ispresented. The brightness or contrast may be reduced when the viewingdirection is towards the sun.

A system and method according to the principles of the presentinventions may involve presenting augmented reality content in anenvironment without relying on real objects in the environment or inenvironments without real objects. This may involve receiving ageospatial location, including altitude, of virtual content within anenvironment to understand where the virtual content is to berepresented. It may also involve creating a content anchor point at thegeospatial location. The system and method may further involve receivingsensor information from a real aircraft sensor system affixed to a realaircraft to provide a location of the aircraft including an altitude ofthe aircraft, speed of the aircraft, and directional attitude of theaircraft and receiving head position information identifying a viewingposition of a pilot within the aircraft. With the virtual contentlocation anchor point understood and the location and conditions of thereal aircraft understood, augmented reality content may be presented ina see-through computer display worn by the pilot when the aircraftsensor data, helmet position data and content anchor point alignindicating the pilot sees the anchor point.

A system and method according to the principles of the presentinventions may involve two or more real airplanes operating in a commonvirtual environment where the pilot's of the respective airplane's arepresented common augmented reality content from each's respectiveperspectives. In embodiments, a computer product, operating on one ormore processors, adapted to present augmented reality content to aplurality of aircraft within a common virtual environment may include adata transmission system adapted to receive geospatial location datafrom the plurality of aircraft, wherein each of the plurality ofaircraft is within visual proximity of one another. It may furtherinvolve a training simulation system adapted to generate a contentanchor at a geospatial location within visual proximity of the pluralityof aircraft in an environment. A content presentation system may beadapted to present computer-generated content representing a trainingasset moving within the visual proximity of the plurality of aircraft toeach of the plurality of aircraft such that a pilot in each respectiveaircraft sees the computer-generated content at a perspective determinedat least in part on the respective aircraft's location with respect tothe anchor location.

A system and method according to the principles of the presentinventions may involve two or more real airplanes operating in a commonvirtual environment where the pilot's of the respective airplane's arepresented common augmented reality content from each's respectiveperspectives. In embodiments, a computer product, operating on one ormore processors, adapted to present augmented reality content to aplurality of aircraft within a common virtual environment may include adata transmission system adapted to receive geospatial location datafrom the plurality of aircraft, wherein each of the plurality ofaircraft is geographically separated such that they cannot see oneanother. Even though they cannot see one another, the training exerciseand virtual environment may be configured such that they are virtuallyin close proximity. Each pilot may be able to ‘see’ the other plane byseeing an augmented reality representation of the other plane. It mayfurther involve a training simulation system adapted to generate acontent anchor at a geospatial location within visual proximity of theplurality of aircraft in an environment. A content presentation systemmay be adapted to present computer-generated content representing atraining asset moving within the visual proximity of the plurality ofaircraft to each of the plurality of aircraft such that a pilot in eachrespective aircraft sees the computer-generated content at a perspectivedetermined at least in part on the respective aircraft's location withrespect to the anchor location.

A system and method according to the principles of the presentinventions may involve a simulated training environment with a movinganchor point for virtual content representing a moving augmented realitytraining asset. In embodiments, a computer product, operating on one ormore processors, may be adapted to present augmented reality content toa pilot of an aircraft. A data transmission system may be adapted toreceive geospatial location data from the aircraft as it moves throughan environment. A training simulation system may be adapted to generatea series of content anchors at geospatial locations within visualproximity of the aircraft, each of the series of content anchorsrepresenting a geospatial position of a virtual training asset movingthrough the environment. A content presentation system may be adapted topresent the virtual training asset to the aircraft such that a pilot inthe aircraft sees the virtual training asset when it is indicated thatthe pilot viewing angle is aligned with a content anchor from the seriesof content anchors that represents a then current location of thevirtual training asset. The virtual training asset is shaped in aperspective view consistent with the pilot's viewing angle and the thencurrent location of the virtual training asset. For example, a series ofprogressively changing geospatial locations may represent a movement ofa virtual training asset through a virtual environment over a period oftime. The movement may be prescribed or pre-programmed and it mayrepresent a sub-second period of time, second(s) period of time,minute(s) period of time, etc. The time period may represent a futureperiod of time to describe how the virtual training asset is going tomove in the future. When it becomes time to present the content to theaugmented reality system in the aircraft the content may be located atone of the series of locations that represents the then current time toproperly align the content. In embodiments, the selected location fromthe series of locations may be a time slightly in the future of the thencurrent time to make an accommodation for latency in presenting thecontent.

A system and method according to the principles of the presentinventions may involve a simulated training system where a virtual assethas a geospatial location that is independent of a real aircraft'slocation that is involved in the training. A system and method ofpresenting the simulated training exercise to a pilot in a real aircraftmay involve generating a virtual environment that includes an indicationof where the real aircraft is located and what its positional attitudeis within the aircraft's real environment. It may further involvegenerating, within the virtual environment, a virtual asset that iswithin a visual range of the real aircraft's location and presenting thevirtual asset to the pilot as augmented reality content that overlaysthe pilot's real view of the environment outside of the real aircraft,wherein the virtual asset is presented at a geospatial position that isindependent of the real aircraft's location. In embodiments, the virtualasset may move in relation to the aircraft's location and maintain thevirtual asset's autonomous movement and location with respect to theaircraft's location. While the virtual asset may react to the realaircraft's movements, the virtual asset may maintain its autonomouscontrol.

The inventors discovered that predicting the future location(s) of areal vehicle that is moving through a real environment can improve theaccuracy of the positioning of virtual content in an augmented realitysystem. This may be especially important when the real vehicle is movingquickly. A system and method in accordance with the principles of thepresent inventions may involve receiving a series of progressivelychanging content geospatial locations representing future movement of avirtual asset within a virtual environment, which may be predeterminedand preprogrammed. It may also involve receiving a series ofprogressively changing real vehicle geospatial locations, eachassociated with a then current acquisition time, representing movementof a real vehicle in a real environment, wherein the virtual environmentgeospatially represents the real environment. The system and method maypredict, based on the series of vehicle locations and relatedacquisition times, a future geospatial location, and series of futurelocations, of the vehicle. Then the augmented reality content may bepresented to an operator of the vehicle at a position within afield-of-view of a see-through computer display based on the futuregeospatial location of the vehicle, or a location from the series oflocations. It may further be based on the geospatial location of thevirtual content, from the series of progressively changing contentgeospatial locations, representative of a time substantially the same asa time represented by the future geospatial location.

In embodiments, the prediction of the future geospatial location of thevehicle may be based at least in part on past geospatial vehiclelocations identified by a sensor system affixed to the vehicle thatperiodically communicates a then current geospatial location; whereinthe past geospatial vehicle locations are interpolated to form a pastvehicle location trend. The prediction of the future geospatial locationof the vehicle may then be further based on an extrapolation based atleast in part on the past vehicle trend. The vehicle may be furtherrepresented by an attitude within the real environment and the virtualasset is represented by an attitude within the virtual environment andthe presentation of the augmented reality content is further based onthe attitude of the vehicle and the attitude of the virtual asset.

A system according to the principles of the present disclosure tracks anairplane's geospatial location (e.g. through GPS) as it moves throughthe air. It also tracks inertial movements of the plane as well as theavionics in the plane; such as pilot controls for thrust, rudder,alerions, elevator, thrust direction, compass, airspeed indicator,external temperature, g-force meter, etc. With this data, a processor,either onboard or off-plane, can determine an accurate understanding ofthe plane's current condition, location, attitude, speed, etc. Suchprocessed data can be tracked over time such that a trend analysis canbe performed on the data in real time. This real time trend analysis canfurther be used to predict where the plane is going to be at a futurepoint in time. For example, the plane's data may be collected every 4 msand a saved data set may include thousands of points representing theimmediate past. The data set can then be used to accurately predictwhere the plane is going to be in the relative near future (e.g. in thenext milliseconds, seconds, minutes). The extrapolated future locationprediction based on the past data gets less precise the further into thefuture the prediction is making. However, the augmented reality contentis being presented to a see-through optic at a fast refresh rate suchthat the position of the content in the optic can be based on themillisecond or second level predictions. As a further example, therefresh rate from a software product that is generating and producingthe virtual content rendering (e.g. a gaming engine) may be on the orderof 4 ms to 12 ms. This means that the position of the content can beshifted to accommodate a predicted location and pilot visions directionevery 4 ms to 12 ms. Knowing the plane's weight and performancecharacteristics may also be used in the calculations. For example, theprocessor may factor in that an F-22 fighter jet weight just over 40,000pounds and can make a 5G turn at 1,000 miles per hour and understandwhat the flight path of such a maneuver may look like. Such flight pathcharacteristics would be quite different in an F-16, Harrier, F-35,Cargo plane, etc.

In embodiments, a system may be equipped with a computer processor toread sensor data from the vehicle (e.g. airplane, ground vehicle, spacevehicle, etc.) to locate the vehicle and understand its currentconditions (e.g. forces, avionics, environment, attitude, etc.). Theprocessor may store the sensor data and evaluate the sensor data. Thetype of vehicle and/or its powered movement characteristics may bestored and used in conjunction with the sensor data to furtherunderstand the present condition of the vehicle. The current and pastsensor data and movement characteristics may be fused and analyzed tounderstand the past performance of the vehicle and this trend analysismay be further used to predict a future position of the vehicle. Withthe very near future position of the vehicle predicted with precision,virtual content can be presented to the see-through optical system usedby a user such that it aligns with a geospatial location of geospatiallylocated content. For example, when the system predicts a location of anairplane one second from now it will be a very accurate prediction. Withthe accurate prediction of the future location and knowing the futuregeospatial positioning of the content (e.g. longitude, latitude, andaltitude) the virtual content can be positioned relative to the positionof the airplane at the future time. The relative, or near absolute,positioning of the content can be refreshed at a very fast rate (e.g. 4ms). This is fast enough to accommodate the fast repositioning of thefast reposition of the virtual content (e.g. another plane approachingfrom the opposite direction).

The inventors further discovered that the head and/or eye position ofthe operator or passenger of the vehicle needs to be well understood asit relates to the position of the vehicle. For example. with an airplanemoving at 1,000 miles an hour and its location and condition wellunderstood (as described herein) it is not enough to determine therelative position of the geospatial content. The content needs to bepresented in the see-through optic at a correct position such that theuser perceives that it as being in the proper geospatial position. In asystem where the see-through optic is attached to the vehiclesurrounding the user's view of the exterior environment, the relativepositioning of the content may require an understanding of the user'seye height since the optic is not moving relative to the vehicle. In asystem where the see-through optic is attached to the user (e.g. headmounted display (“HMD”), in a helmet, etc.) the position of the user'shead will be considered. For example, if the virtual content is on theright side of the vehicle and the user is looking out the left side ofthe vehicle, the content should not be presented to the see-throughoptic because the user cannot see the geospatial location anchoring thecontent. As the user turns her head to view the anchor point the contentwill be presented at a location within the optic that correlates with avirtual line connecting her position within the vehicle and the anchorposition.

In embodiments, the user's head position may be derived using aninside-out (e.g. where an HMD emits electromagnetic energy to measuredistances to objects within a user environment and then determiningposition through triangulation), outside-in (e.g. where there areelectromagnetic energy emitters set at known locations within the user'senvironment and use distance measurements from the emitters to the HMDto triangulate), mechanical system, electrical system, wireless system,wired system, etc. For example, an outside-in system in a cockpit of ajet fighter may use electromagnetics to triangulate the head positionusing emitters located at known positions within the cockpit. The helmetor other HMD may have markers or be markerless. Marks on the helmet maybe used to identify the user's direction of vision. A markerless HMD maybe programmed to understand the electromagnetic signature of the HMDsuch that its viewing position can be derived.

A system may also include an eye tracking system to identify thedirection of the user's eyes. This can be used in conjunction with thehead position data to determine the general direction the user islooking (e.g. through head position tracking) and specific direction(e.g. through eye position). This may be useful in conjunction with afoveated display where the resolution of the virtual content isincreased in the specific direction and decreased otherwise. The acuityof the human eye is very high within a very narrow angle (e.g. 1 or 2degrees) and it quickly falls off outside of the narrow angle. This canmean that content outside of the high acuity region can be decreased inresolution or sharpness because it is going to be perceived as‘peripheral vision’ and it can save processing power and decreaselatency because potentially less data is used to render and presentcontent.

In embodiments, an augmented reality system used by an operator of avehicle may make a precision prediction of the vehicle's futuregeospatial location, orientation, angular position, attitude, direction,speed (this collection of attributes or sub set of attributes or otherattributes describing the vehicle within an environment is generallyreferred to as the vehicle's condition herein), and acceleration basedon the vehicle's past performance of the same factors, or subset orother set of factors, leading up to the vehicle's current state.Including an understanding of the vehicle's capabilities and abilitiesthroughout a range of motions, speeds, accelerations, etc. can assist inthe future prediction. Such an augmented reality system may employartificial intelligence, machine language and the like to make theprediction based on such data collected over time. Such system mayfurther include an error prediction and include limits on how much erroris tolerable given the current situation. For example, the augmentedreality system may be able to predict the future position and geometrywith great accuracy for three second in the future. At a frame rate of10 ms that means three hundred frames of virtual content can be ‘lockedin’ as to its location and geometry. If the prediction after threeseconds and less than five second, for example, is reasonablypredictable, the frames to be generated in that period may be renderedfrom one perspective (e.g. the geometry may be fixed) but not ‘lockedin’ from another (e.g. the location may be approximate to be updatedwhen it gets to the three second prediction point in the data stream.This means you could have three hundred frames locked in and completelyavailable for presentation along with another two hundred frames thatare partially rendered in some way. Optional rendering could also beproduced if the future prediction system developed more than onealternative path for the vehicle. A method allowing the future renderingof content within a gaming engine could reduce the latency of presentingthe content to the see-through optic.

The future location/geometric position/condition prediction systemsdescribed herein are very useful when used in fast moving vehicles. Ajet aircraft may travel at speeds of 1,300 miles per hour. That isequivalent to 1.9 feet per millisecond. If the content rendering systemhas a content data output rate of 10 ms, that means there could be 19feet travelled between frames. That could lead to significantmisplacement or poor rendering of the geometry, orientation, etc. of thevirtual content if a future prediction of the vehicle's location,geometric position, and condition is not used to impact the generationof the content. Even at much slower speeds the error produced withoutthe future prediction may be significant. Cutting the speed down from1300 miles per hour to 130 miles per hour could still lead to a neartwo-foot error between frames in content rendering and placement. Evenat highway speed of 65 miles per hour, a one-foot error could beproduced.

The future prediction of the vehicle's location and condition may bemade to provide processing time before presenting the virtual content.It may further be made such that when the content is ready forpresentation the content can be positioned properly within thesee-through optic.

An augmented reality system and method in accordance with the principlesof the present disclosure may include a geospatial location systemadapted to identify a current location of a vehicle (e.g. GPS), aplurality of sensors adapted to identify the vehicle's positionalgeometry within an environment (e.g. inertial measurement unit (IMU),G-Force sensor, compass) at the current location, a plurality of sensorsadapted to identify vectors of force being applied to the vehicle (e.g.IMU, G-Force sensor); a data association and storage module (e.g. acomputer processor with memory) adapted to associate and store thegeospatial location data, positional geometry data, and force vectordata with a time of acquisition of each type of data, a computerprocessor adapted to analyze the stored data and generate a trend of thevehicle's positions and conditions over a period of time and extrapolatethe trend into a future period of time to produce a future predictedperformance, wherein the processor is further adapted (e.g. programmedto execute) to present geospatially located augmented reality content toan operator of the vehicle based on the future predicted performance.The presentation of content based on the future predicted performance isestimated to be presented at a time corresponding with the then currenttime and location. In other words, the future prediction is used todetermine the location and condition of the vehicle in the future, andpresentation of the content is done using the prediction of location andcondition that is timestamped with the then current time or nearest thencurrent time.

The system and method may further include a head position trackingsystem adapted to identify a viewing direction of a user of an augmentedreality see-through computer display, wherein the presentation of thegeospatially located content is further based on the viewing directionof the user. The presentation of the geospatially located content mayalso involve positioning the content within a field-of-view of thesee-through computer display based on the viewing direction of the user.The system and method may further comprise an eye direction detectionsystem (e.g. a camera system or other sensor system for imaging andtracking the position and movement of the user's eyes, wherein thepresentation of the geospatially located content within thefield-of-view is further based on a measured eye position, direction, ormotion of the user.

FIG. 1 is an illustration of an online platform 100 consistent withvarious embodiments of the present disclosure. By way of non-limitingexample, the online platform 100 to allow real pilots in real aircraftusing augmented and virtual reality to meet in a virtual piece ofairspace may be hosted on a centralized server 102, such as, forexample, a cloud computing service. The centralized server 102 maycommunicate with other network entities, such as, for example, anaugmented and virtual reality display device 106, a sensor system 110 ofan aircraft (such as an aircraft 200, as shown in FIG. 2), database 114(such as, 3D model database), over a communication network 104, such as,but not limited to, the Internet. Accordingly, in some instances, theaugmented and virtual reality display device 106 operated by a pilot (auser 112) may be in communication with the online platform 100. Further,the sensor system 110 of the aircraft 200 may be in communication withthe online platform 100. All communication between the augmented andvirtual reality display device 106 and the sensor system 110 with theonline platform 100 may be carried via radio waves. For example,Aircraft Communications Addressing and Reporting System (ACARS) may beused for communication between the augmented and virtual reality displaydevice 106 or the sensor system 110, and the online platform 100.

Further, the centralized server 102 may include one or more servers; forexample, a master server and one or more local servers. The one or moreservers may be stationed on one or more of the aircraft, the ground anda satellite orbiting the earth (such as Satcom and Iridium satellites).Further, as shown in FIG. 2, the aircraft 200 may include a RemoteArtificial Intelligence Link (RAIL) 202 for communication with thecentralized server 102. Further, the AI-driven processing and thegraphics generation may be performed on the centralized server 102.

The augmented and virtual reality display device 106 may display contentto a pilot flying the aircraft 200. The augmented and virtual realitydisplay device 106 may be one of a head-mounted display (HMD),Eyeglasses, head-up display (HUD), smart contact lenses, Virtual retinaldisplay, EyeTap, and cockpit glass. In some embodiments, the augmentedand virtual reality display device 106 may be integrated with a flighthelmet of a pilot. As shown in FIG. 2, an Enhanced Visual Environment(EVE) 204 may be configured to provide high fidelity/wide field of viewcontent to the augmented and virtual reality display device 106.

The sensor system 110 of the aircraft 200 may include one or moreinternal sensors to track and localize the pilot's head within thecockpit of the aircraft 200. Further, the sensor system 110 of theaircraft 200 may include one or more external sensors to track theposition and orientation of the aircraft 200. As shown in FIG. 2, anAvionics Integration System (AIS) 206 may be configured to provideaccurate six degrees of freedom positioning of aircraft 200. The sixdegrees of freedom include longitudinal (forward and backward thrust),vertical (aircraft moves upward and downward), lateral (aircraft movesfrom side to side), pitch (nose pitches up or down), roll (wings roll upor down) and yaw (nose moves from side to side).

Data fusion can be an important feature of systems according toembodiments of this disclosure. A processor may need to read data fromseveral different sources in the process of determining a vehicle'scurrent location and condition. The processor may further send datarepresenting the predictions to a content rendering system. Theprocessor may further have to receive renderings from the renderingsystem and then present the rendering to an HMD at the right time tomatch the then current position and condition of the vehicle. This maybe referred to as data fusion. To make the timing of the presentation ofcontent even more complicated, as the inventors further discovered, dataupon which a location/condition prediction might be made may haverefresh rates that may be different and the content rendering refreshrate may be different and any of it may possibly have variable refreshrates.

In embodiments, the augmented reality system, through a processor,produces a prediction of the future location and condition of a vehicleover a future time period. The future time period may include discretedata points at discrete time intervals. The intervals may be coordinatedwith the incoming data refresh rates. The processor may interpolate databetween the discrete points in time to provide a higher resolutionprediction. This may be useful in situations where the rendering enginehas a variable or different refresh rate from the data being used forthe vehicle's predicted future position and condition. For example, thedata used to predict the location and condition of the vehicle may havea refresh rate of 5 ms and the rendering engine may have a variablerefresh rate of between 4 and 12 ms. The processor might theninterpolate between the discrete future positions and conditions suchthat when the content does arrive for presentation it knows thevehicle's then current predicted state in a predictable resolution.

In embodiments, a processor may interpolate each data type within itsown refresh rate such that several data types with different refreshrates can be merged at common timestamps. The merged data may then beanalyzed to generate a trend of the vehicle's locations and conditions.This trend may be analyzed and extrapolated to predict future locationsand conditions of the vehicle.

Further, as shown in FIG. 2, Coupled Fusion Tracking (CFT) 208 may beemployed to combine the data received from the one or more internalsensors and the one or more external sensors to provide a highly usableaugmented reality solution in a fast-moving environment. Further, theCFT 208 may integrate both virtual reality and augmented reality toprovide robust augmented reality visuals within a dynamic environment.For example, the CFT 208 may allow for drawing an accurate picture of anenemy aircraft in augmented and virtual reality display device 106 wornby a pilot.

The user 112 may access online platform 100 through any useful userinterface (e.g. software application, browser, etc. The softwareapplication may be embodied as, for example, but not be limited to, asoftware user interface, network interface, website, web application,desktop application, augmented reality application, virtual realityapplication, mobile application compatible with a computing device 1600,etc.

Systems and methods described herein may be used to provide a commonvirtual environment to more than one person in more than one vehicle.This may be referred to generally as a common virtual environment,coordinated virtual environment or virtual environment. A common virtualenvironment may or may not be presented in a single geospatialenvironment. When two jet fighters are practicing maneuvers together inthe same airspace (e.g. where the pilots of two planes have the abilityto see one another) the common virtual environment may represent theairspace they occupy. If, on the other hand, two jet fighters arepracticing maneuvers in separate airspaces (e.g. where the two pilotsnever actually see one another) the common virtual environment mayrepresent one of their airspaces, neither of their airspaces, etc. Ineach case, the presentation of virtual content in the common virtualenvironment may involve understanding the geospatial location andcondition of each plane such that the common virtual environment can bepresented from the correct perspective in each plane.

There could be many real vehicles and people within a common virtualenvironment. There may be a number of planes, ground vehicles and peopleparticipating in a training session, game, or other activity. Each wouldbe seeing the common virtual environment from their own perspectivethrough their own HMD. Each vehicle may have a system as describedherein to track the vehicles' location and condition to predict itsfuture location and condition for the placement of virtual content inits associated see-through optical system. Further, each vehicle and/orHMD may have a head and/or eye tracking system upon which the contentposition may in part depend.

Systems and methods according to the principles of the present inventionmay involve training a plurality of pilots, each in a separate realaircraft, where the plurality of separate aircraft share a commonphysical environment. This may be useful in a training situation wheretwo or more planes are flying in close proximity and are being presentedwith a common enemy asset in augmented reality. This could be a dogfight, missile aversion, target bombing, etc. Such systems and methodsmay include providing a head mounted see-through computer display (HMD)to each of the plurality of pilots such that each of the plurality ofpilots can view a common virtual environment with computer renderedtraining content. Each of the aircraft may track and report its ownlocation, attitude, speed, or other information to a computer simulationsystem such that the simulation system can manage the trainingsimulation. The simulation system may position the computer renderedtraining content at a geospatial location within a visual range of eachof the plurality of pilots or one of the pilots and the content may bepresented to the HMD of each of the plurality of pilots, wherein thepresentation in each individual HMD is dependent on an alignment of eachrespective HMD and the computer rendered content geospatial location.

In embodiments, the computer rendered training content presented to eachHMD is rendered with its own unique perspective based on the angle fromwhich each HMD views the geospatial location. In this example, each ofthe plurality of pilots has the ability to see another of the pluralityof aircraft through their HMD, forming an augmented reality trainingenvironment comprising a see through view of the real environment foreach pilot augmented by the computer rendered training content presentedin the common virtual environment. Each of the plurality of pilots maybe in communication with the other pilots such that the pilots cannavigate their separate real aircraft in coordination in response to thecomputer rendered training content.

Systems and methods according to the principles of the present inventionmay involve presenting a plurality of pilots of separate aircraft with acommon augmented reality environment where common computer generatedcontent is positioned and each of the plurality of pilots sees thecommon computer generated content from at a perspective based on theirrespective locations and aircraft's attitude. Each of the pilots may beable to communicate with the other pilots such that they can coordinatenavigation maneuvers with respect to the computer generated content.

In embodiments, the computer generated content may be a representationof an enemy asset, wherein the enemy asset is programmed to engage withat least one of the separate aircraft. The computer generated contentmay represent a plurality of independently controlled enemy assets,wherein each of the a plurality of independently controlled enemy assetis programmed to engage with at least one of the separate aircraft. Thismay simulate a coordinated enemy, which may require team navigation andcoordination.

In embodiments, the presentation of the computer generated content toeach of the plurality of pilots may be based on an alignment betweeneach of the plurality of pilots viewing direction and the computergenerated content's geospatial location such that each pilot sees thecomputer generated content when each pilot's aircraft position, pilotviewing direction and the content's geospatial location align in anunobstructed line of sight. For example, if a plane is flying level andwithin visual range of the content's geospatial location, the pilot maysee the content if it is in front of the plane and above the planehorizon such that the pilot can see the geospatial location through thecockpit window. If, on the other hand, the content is directly behindthe plane and the pilot cannot turn his head to view the geospatiallocation of the content, than the content may not be presented in thepilot's HMD.

FIG. 3 is a block diagram of a system 300 for facilitating provisioningof a virtual experience in accordance with some embodiments. The system300 may include a communication device 302, a processing device 304 anda storage device 306.

The communication device 302 may be configured for receiving at leastone first sensor data corresponding to at least one first sensor 310associated with a first vehicle 308. Further, the at least one firstsensor 310 may be communicatively coupled to a first transmitter 312configured for transmitting the at least one first sensor data over afirst communication channel. In some embodiments, the first vehicle 308may be a first aircraft. Further, a first user may be a first pilot.

Further, the communication device 302 may be configured for receiving atleast one second sensor data corresponding to at least one second sensor320 associated with a second vehicle 318. Further, the at least onesecond sensor 320 may be communicatively coupled to a second transmitter322 configured for transmitting the at least one second sensor data overa second communication channel. In some embodiments, the second vehicle318 may be a second aircraft. Further, a second user may be a secondpilot.

In some embodiments, the at least one first sensor data may be receivedfrom a first On-Board-Diagnostics (OBD) system of the first vehicle 308,the at least one second sensor data may be received from a secondOn-Board-Diagnostics (OBD) system of the second vehicle 318.

Further, the communication device 302 may be configured for transmittingat least one first presentation data to at least one first presentationdevice 314 associated with the first vehicle 308. Further, the at leastone first presentation device 314 may include a first receiver 316configured for receiving the at least one first presentation data overthe first communication channel. Further, the at least one firstpresentation device may be configured for presenting the at least onefirst presentation data.

Further, the communication device 302 may be configured for transmittingat least one second presentation data to at least one secondpresentation device 324 associated with the second vehicle 318. Further,the at least one second presentation device 324 may include a secondreceiver 326 configured for receiving the at least one secondpresentation data over the second communication channel. Further, the atleast one second presentation device may be configured for presentingthe at least one second presentation data.

Further, the processing device 304 may be configured for generating theat least one first presentation data based on the at least one secondsensor data.

Further, the processing device 304 may be configured for generating theat least one second presentation data based on the at least one firstsensor data.

Further, the storage device 306 may be configured for storing each ofthe at least one first presentation data and the at least one secondpresentation data.

In some embodiments, the at least one first sensor 310 may include oneor more of a first orientation sensor, a first motion sensor, a firstaccelerometer, a first location sensor, a first speed sensor, a firstvibration sensor, a first temperature sensor, a first light sensor and afirst sound sensor. Further, the at least one second sensor 320 mayinclude one or more of a second orientation sensor, a second motionsensor, a second accelerometer, a second location sensor, a second speedsensor, a second vibration sensor, a second temperature sensor, a secondlight sensor and a second sound sensor.

In some embodiments, the at least one first sensor 310 may be configuredfor sensing at least one first physical variable associated with thefirst vehicle 308. Further, the at least one second sensor 320 may beconfigured for sensing at least one second physical variable associatedwith the second vehicle. In further embodiments, the at least one firstphysical variable may include one or more of a first orientation, afirst motion, a first acceleration, a first location, a first speed, afirst vibration, a first temperature, a first light intensity and afirst sound. Further, the at least one second physical variable mayinclude one or more of a second orientation, a second motion, a secondacceleration, a second location, a second speed, a second vibration, asecond temperature, a second light intensity and a second sound.

In some embodiments, the at least one first sensor 310 may include afirst environmental sensor configured for sensing a first environmentalvariable associated with the first vehicle 308. Further, the at leastone second sensor 320 may include a second environmental sensorconfigured for sensing a second environmental variable associated withthe second vehicle.

In some embodiments, the at least one first sensor 310 may include afirst user sensor configured for sensing a first user variableassociated with a first user of the first vehicle 308. Further, the atleast one second sensor 320 may include a second user sensor configuredfor sensing a second user variable associated with a second user of thesecond vehicle 318.

In further embodiments, the first user variable may include a first userlocation and a first user orientation. Further, the second user variablemay include a second user location and a second user orientation.Further, the first presentation device may include a first head mountdisplay. Further, the second presentation device may include a secondhead mount display.

In further embodiments, the first head mount display may include a firstuser location sensor of the at least one first sensor 310 configured forsensing the first user location and a first user orientation sensor ofthe at least one first sensor 310 configured for sensing the first userorientation. The first head mount display is explained in further detailin conjunction with FIG. 4 below. Further, the second head mount displaymay include a second user location sensor of the at least one secondsensor 320 configured for sensing the second user location, a seconduser orientation sensor of the at least one second sensor 320 configuredfor sensing the second user orientation.

In further embodiments, the first vehicle 308 may include a first userlocation sensor of the at least one first sensor 310 configured forsensing the first user location and a first user orientation sensor ofthe at least one first sensor 310 configured for sensing the first userorientation. Further, the second vehicle 318 may include a second userlocation sensor of the at least one second sensor 320 configured forsensing the second user location, a second user orientation sensor ofthe at least one second sensor 320 configured for sensing the seconduser orientation.

In further embodiments, the first user orientation sensor may include afirst gaze sensor configured for sensing a first eye gaze of the firstuser. Further, the second user orientation sensor may include a secondgaze sensor configured for sensing a second eye gaze of the second user.

In further embodiments, the first user location sensor may include afirst proximity sensor configured for sensing the first user location inrelation to the at least one first presentation device 314. Further, thesecond user location sensor may include a second proximity sensorconfigured for sensing the second user location in relation to the atleast one second presentation device 324.

In some embodiments, the first head mount display may include a firstsee-through display device. Further, the second head mount display mayinclude a second see-through display device.

In some embodiments, the first head mount display may include a firstoptical marker configured to facilitate determination of one or more ofthe first user location and the first user orientation. Further, the atleast one first sensor 310 may include a first camera configured forcapturing a first image of the first optical marker. Further, the atleast one first sensor 310 may be communicatively coupled to a firstprocessor associated with the vehicle. Further, the first processor maybe configured for determining one or more of the first user location andthe first user orientation based on analysis of the first image.Further, the second head mount display may include a second opticalmarker configured to facilitate determination of one or more of thesecond user location and the second user orientation. Further, the atleast one second sensor 320 may include a second camera configured forcapturing a second image of the second optical marker. Further, the atleast one second sensor 320 may be communicatively coupled to a secondprocessor associated with the vehicle. Further, the second processor maybe configured for determining one or more of the second user locationand the second user orientation based on analysis of the second image.

In some embodiments, the first presentation device may include a firstsee-through display device disposed in a first windshield of the firstvehicle 308. Further, the second presentation device may include asecond see-through display device disposed in a second windshield of thesecond vehicle 318.

In some embodiments, the first vehicle 308 may include a firstwatercraft, a first land vehicle, a first aircraft and a firstamphibious vehicle. Further, the second vehicle 318 may include a secondwatercraft, a second land vehicle, a second aircraft and a secondamphibious vehicle.

In some embodiments, the at least one first presentation data mayinclude one or more of a first visual data, a first audio data and afirst haptic data. Further, the at least one second presentation datamay include one or more of a second visual data, a second audio data anda second haptic data.

In some embodiments, the at least one first presentation device 314 mayinclude at least one environmental variable actuator configured forcontrolling at least one first environmental variable associated withthe first vehicle 308 based on the first presentation data. Further, theat least one second presentation device 324 may include at least oneenvironmental variable actuator configured for controlling at least onesecond environmental variable associated with the second vehicle 318based on the second presentation data. In further embodiments, the atleast one first environmental variable may include one or more of afirst temperature level, a first humidity level, a first pressure level,a first oxygen level, a first ambient light, a first ambient sound, afirst vibration level, a first turbulence, a first motion, a firstspeed, a first orientation and a first acceleration, the at least onesecond environmental variable may include one or more of a secondtemperature level, a second humidity level, a second pressure level, asecond oxygen level, a second ambient light, a second ambient sound, asecond vibration level, a second turbulence, a second motion, a secondspeed, a second orientation and a second acceleration.

In some embodiments, the first vehicle 308 may include each of the atleast one first sensor 310 and the at least one first presentationdevice 314. Further, the second vehicle 318 may include each of the atleast one second sensor 320 and the at least one second presentationdevice 324.

In some embodiments, the storage device 306 may be further configuredfor storing a first three-dimensional model corresponding to the firstvehicle 308 and a second three-dimensional model corresponding to thesecond vehicle 318. Further, the generating of the first presentationdata may be based on the second three-dimensional model. Further, thegenerating of the second presentation data may be based on the firstthree-dimensional model.

In some embodiments, the communication device 302 may be furtherconfigured for receiving an administrator command from an administratordevice. Further, the generating of one or more of the first presentationdata and the second presentation data may be based on the administratorcommand. In further embodiments, the at least one first presentationdata may include at least one first virtual object model correspondingto at least one first virtual object. Further, the at least one secondpresentation data may include at least one second virtual object modelcorresponding to at least one second virtual object. Further, thegenerating of the at least one first virtual object model may beindependent of the at least one second sensor model. Further, thegenerating of the at least one second virtual object model may beindependent of the at least one first sensor model. Further, thegenerating of one or more of the at least one first virtual object modeland the at least one second virtual object model may be based on theadministrator command. Further, the storage device 306 may be configuredfor storing the at least one first virtual object model and the at leastone second virtual object model.

In further embodiments, the administrator command may include a virtualdistance parameter. Further, the generating of each of the at least onefirst presentation data and the at least one second presentation datamay be based on the virtual distance parameter.

In further embodiments, the at least one first sensor data may includeat least one first proximity data corresponding to at least one firstexternal real object in a vicinity of the first vehicle 308. Further,the at least one second sensor data may include at least one secondproximity data corresponding to at least one second external real objectin a vicinity of the second vehicle 318. Further, the generating of theat least one first presentation data may be based on the at least onesecond proximity data. Further, the generating of the at least onesecond presentation data may be based on the at least one firstproximity data. In further embodiments, the at least one first externalreal object may include a first cloud, a first landscape feature, afirst man-made structure and a first natural object. Further, the atleast one second external real object may include a second cloud, asecond landscape feature, a second man-made structure and a secondnatural object.

In some embodiments, the at least one first sensor data may include atleast one first image data corresponding to at least one first externalreal object in a vicinity of the first vehicle 308. Further, the atleast one second sensor data may include at least one second image datacorresponding to at least one second external real object in a vicinityof the second vehicle 318. Further, the generating of the at least onefirst presentation data may be based on the at least one second imagedata. Further, the generating of the at least one second presentationdata may be based on the at least one first image data.

In some embodiments, the communication device 302 may be furtherconfigured for transmitting a server authentication data to the firstreceiver 316. Further, the first receiver 316 may be communicativelycoupled to first processor associated with the first presentationdevice. Further, the first processor may be communicatively coupled to afirst memory device configured to store a first authentication data.Further, the first processor may be configured for performing a firstserver authentication based on the first authentication data and theserver authentication data. Further, the first processor may beconfigured for controlling presentation of the at least one firstpresentation data on the at least one first presentation device 314based on the first server authentication. Further, the communicationdevice 302 may be configured for transmitting a server authenticationdata to the second receiver 326. Further, the second receiver 326 may becommunicatively coupled to second processor associated with the secondpresentation device. Further, the second processor may becommunicatively coupled to a second memory device configured to store asecond authentication data. Further, the second processor may beconfigured for performing a second server authentication based on thesecond authentication data and the server authentication data. Further,the second processor may be configured for controlling presentation ofthe at least one second presentation data on the at least one secondpresentation device 324 based on the second server authentication.Further, the communication device 302 may be configured for receiving afirst client authentication data from the first transmitter 312.Further, the storage device 306 may be configured for storing the firstauthentication data. Further, the communication device 302 may beconfigured for and receiving a second client authentication data fromthe second transmitter 322. Further, the storage device 306 may beconfigured for storing the second authentication data. Further, theprocessing device 304 may be further configured for performing a firstclient authentication based on the first client authentication data andthe first authentication data. Further, the generating of the at leastone second presentation data may be further based on the first clientauthentication. Further, the processing device 304 may be configured forperforming a second client authentication based on the second clientauthentication data and the second authentication data. Further, thegenerating of the at least one first presentation data may be furtherbased on the second client authentication.

FIG. 4 is a block diagram of a first head mount display 400 forfacilitating provisioning of a virtual experience in accordance withsome embodiments. The first head mount display 400 includes a first userlocation sensor 402 of the at least one first sensor configured forsensing the first user location and a first user orientation sensor 404of the at least one first sensor configured for sensing the first userorientation.

Further, the first head mount display 400 may include a display device406 to present visuals. The display device may a first see-throughdisplay device.

Further, the first head mount display 400 may include a processingdevice 408 configured to obtain sensor data from the first user locationsensor 402 and the first user orientation sensor 404. Further, theprocessing device 408 may be configured to send visuals to the displaydevice 406.

FIG. 5 is a block diagram of an apparatus 500 for facilitatingprovisioning of a virtual experience in accordance with someembodiments. The apparatus 500 may include at least one first sensor 502(such as the at least one first sensor 310) configured for sensing atleast one first sensor data associated with a first vehicle (such as thefirst vehicle 308). Further, the apparatus 500 may include a firsttransmitter 504 (such as the first transmitter 312) configured to becommunicatively coupled to the at least first sensor 502. Further, thefirst transmitter 504 may be further configured for transmitting the atleast one first sensor data to a communication device (such as thecommunication device 302) of a system over a first communicationchannel.

Further, the apparatus 500 may include a first receiver 506 (such as thefirst receiver 316) configured for receiving the at least one firstpresentation data from the communication device over the firstcommunication channel.

Further, the apparatus 500 may include at least one first presentationdevice 508 (such as the at least one first presentation device 314)configured to be communicatively coupled to the first receiver 506. Theat least one first presentation device 508 may be configured forpresenting the at last one first presentation data.

Further, the communication device may be further configured forreceiving at least one second sensor data corresponding to at least onesecond sensor (such as the at least one second sensor 320) associatedwith a second vehicle (such as the second vehicle 318). Further, the atleast one second sensor may be communicatively coupled to a secondtransmitter (such as the second transmitter 322) configured fortransmitting the at least one second sensor data over a secondcommunication channel. Further, the system further may include aprocessing device (such as the processing device 304) communicativelycoupled to the communication device. Further, the processing device maybe configured for generating the at least one first presentation databased on the at least one second sensor data.

FIG. 6 is a flowchart of a method 600 of facilitating provisioning of avirtual experience in accordance with some embodiments. At 602, themethod 600 may include receiving, using a communication device (such asthe communication device 302), at least one first sensor datacorresponding to at least one first sensor (such as the at least onefirst sensor 310) associated with a first vehicle (such as the firstvehicle 308). Further, the at least one first sensor may becommunicatively coupled to a first transmitter (such as the firsttransmitter 312) configured for transmitting the at least one firstsensor data over a first communication channel.

At 604, the method 600 may include receiving, using the communicationdevice, at least one second sensor data corresponding to at least onesecond sensor (such as the at least one second sensor 320) associatedwith a second vehicle (such as the second vehicle 318). Further, the atleast one second sensor may be communicatively coupled to a secondtransmitter (such as the second transmitter 322) configured fortransmitting the at least one second sensor data over a secondcommunication channel.

At 606, the method 600 may include transmitting, using the communicationdevice, at least one first presentation data to at least one firstpresentation device associated with the first vehicle. Further, the atleast one first presentation device may include a first receiver (suchas the first receiver 316) configured for receiving the at least onefirst presentation data over the first communication channel. Further,the at least one first presentation device may be configured forpresenting the at least one first presentation data.

At 608, the method 600 may include transmitting, using the communicationdevice, at least one second presentation data to at least one secondpresentation device (such as the at least one second presentation device324) associated with the second vehicle. Further, the at least onesecond presentation device may include a second receiver (such as thesecond receiver 326) configured for receiving the at least one secondpresentation data over the second communication channel. Further, the atleast one second presentation device may be configured for presentingthe at least one second presentation data.

At 610, the method 600 may include generating, using a processing device(such as the processing device 304), the at least one first presentationdata based on the at least one second sensor data.

At 612, the method 600 may include generating, using the processingdevice, the at least one second presentation data based on the at leastone first sensor data.

At 614, the method 600 may include storing, using a storage device (suchas the storage device 306), each of the at least one first presentationdata and the at least one second presentation data.

FIG. 7 shows a system 700 for facilitating provisioning of a virtualexperience, in accordance with some embodiments. The system 700 mayinclude a communication device 702 configured for receiving at least onefirst sensor data corresponding to at least one first sensor 710associated with a first vehicle 708. Further, the at least one firstsensor 710 may be communicatively coupled to a first transmitter 712configured for transmitting the at least one first sensor data over afirst communication channel.

Further, the communication device 702 may be configured for receiving atleast one second sensor data corresponding to at least one second sensor716 associated with a second vehicle 714. Further, the at least onesecond sensor 716 may include a second location sensor configured todetect a second location associated with the second vehicle 714.Further, the at least one second sensor 716 may be communicativelycoupled to a second transmitter 718 configured for transmitting the atleast one second sensor data over a second communication channel.Further, in some embodiments, the at least one second sensor 716 mayinclude a second user sensor configured for sensing a second uservariable associated with a second user of the second vehicle 714.Further, the second user variable may include a second user location anda second user orientation.

Further, the communication device 702 configured for transmitting atleast one second presentation data to at least one second presentationdevice 720 associated with the second vehicle 714. Further, the at leastone second presentation data may include at least one second virtualobject model corresponding to at least one second virtual object.Further, in some embodiments, the at least one second virtual object mayinclude one or more of a navigational marker (such as a navigationalmarker 1308, and/or a signboard 1504 as shown in FIG. 15) and anair-corridor (such as a skyway 1306 as shown in FIG. 13). Further, theat least one second presentation device 720 may include a secondreceiver 722 configured for receiving the at least one secondpresentation data over the second communication channel. Further, the atleast one second presentation device 720 may be configured forpresenting the at least one second presentation data. Further, in someembodiments, the at least one second presentation device 720 may includea second head mount display. Further, the second head mount display mayinclude a second user location sensor of the at least one second sensor716 configured for sensing the second user location and a second userorientation sensor of the at least one second sensor 716 configured forsensing the second user orientation. Further, the second head mountdisplay may include a second see-through display device.

Further, the system 700 may include a processing device 704 configuredfor generating the at least one second presentation data based on the atleast one first sensor data and the at least one second sensor data.Further, the generating of the at least one second virtual object modelmay be independent of the at least one first sensor data. Further, insome embodiments, the processing device 704 may be configured fordetermining a second airspace class (with reference to FIG. 14)associated with the second vehicle 714 based on the second locationincluding a second altitude associated with the second vehicle 714.Further, the generating of the at least one second virtual object modelmay be based on the second airspace class.

Further, the system 700 may include a storage device 706 configured forstoring the at least one second presentation data. Further, in someembodiments, the storage device 706 may be configured for retrieving theat least one second virtual object model based on the second locationassociated with the second vehicle 714. Further, in some embodiments,the storage device 706 may be configured for storing a firstthree-dimensional model corresponding to the first vehicle 708. Further,the generating of the second presentation data may be based on the firstthree-dimensional model.

Further, in some embodiments, the communication device 702 may beconfigured for receiving an administrator command from an administratordevice. Further, the generating of the at least one second virtualobject model may be based on the administrator command.

Further, in some embodiments, the communication device 702 may beconfigured for transmitting at least one first presentation data to atleast one first presentation device (not shown) associated with thefirst vehicle 708. Further, the at least one first presentation devicemay include a first receiver configured for receiving the at least onefirst presentation data over the first communication channel. Further,the at least one first presentation device may be configured forpresenting the at least one first presentation data. Further, in someembodiments, the processing device 704 may be configured for generatingthe at least one first presentation data based on the at least onesecond sensor data. Further, in some embodiments, the storage device 706may be configured for storing the at least one first presentation data.Further, in some embodiments, the storage device 706 may be configuredfor storing a second three-dimensional model corresponding to the secondvehicle 714. Further, the generating of the first presentation data maybe based on the second three-dimensional model.

Further, in some embodiments, the at least one first presentation datamay include at least one first virtual object model corresponding to atleast one first virtual object. Further, the generating of the at leastone first virtual object model may be independent of the at least onesecond sensor data. Further, the storage device 706 may be configuredfor storing the at least one first virtual object model.

Further, in some exemplary embodiment, the communication device 702 maybe configured for receiving at least one second sensor datacorresponding to at least one second sensor 716 associated with a secondvehicle 714. Further, the at least one second sensor 716 may becommunicatively coupled to a second transmitter 718 configured fortransmitting the at least one second sensor data over a secondcommunication channel. Further, the communication device 702 may beconfigured for receiving at least one first sensor data corresponding toat least one first sensor 710 associated with a first vehicle 708.Further, the at least one first sensor 710 may include a first locationsensor configured to detect a first location associated with the firstvehicle 708. Further, the at least one first sensor 710 may becommunicatively coupled to a first transmitter 712 configured fortransmitting the at least one first sensor data over a firstcommunication channel. Further, in some embodiments, the at least onefirst sensor 710 may include a first user sensor configured for sensinga first user variable associated with a first user of the first vehicle708. Further, the first user variable may include a first user locationand a first user orientation. Further, the communication device 702configured for transmitting at least one first presentation data to atleast one first presentation device (not shown) associated with thefirst vehicle 708. Further, the at least one first presentation data mayinclude at least one first virtual object model corresponding to atleast one first virtual object. Further, in some embodiments, the atleast one first virtual object may include one or more of a navigationalmarker (such as a navigational marker 1308, and/or a signboard 1504 asshown in FIG. 15) and an air-corridor (such as a skyway 1306 as shown inFIG. 13). Further, the at least one first presentation device mayinclude a first receiver configured for receiving the at least one firstpresentation data over the first communication channel. Further, the atleast one first presentation device may be configured for presenting theat least one first presentation data. Further, in some embodiments, theat least one first presentation device may include a first head mountdisplay. Further, the first head mount display may include a first userlocation sensor of the at least one first sensor 710 configured forsensing the first user location and a first user orientation sensor ofthe at least one first sensor 710 configured for sensing the first userorientation. Further, the first head mount display may include a firstsee-through display device. Further, the processing device 704 may beconfigured for generating the at least one first presentation data basedon the at least one second sensor data and the at least one first sensordata. Further, the generating of the at least one first virtual objectmodel may be independent of the at least one second sensor data.Further, in some embodiments, the processing device 704 may beconfigured for determining a first airspace class (with reference toFIG. 14) associated with the first vehicle 708 based on the firstlocation including a first altitude associated with the first vehicle708. Further, the generating of the at least one first virtual objectmodel may be based on the first airspace class. Further, in someembodiments, the storage device 706 may be configured for storing the atleast one first presentation data. Further, in some embodiments, thestorage device 706 may be configured for retrieving the at least onefirst virtual object model based on the first location associated withthe first vehicle 708. Further, in some embodiments, the storage device706 may be configured for storing a second three-dimensional modelcorresponding to the second vehicle 714. Further, the generating of thefirst presentation data may be based on the second three-dimensionalmodel. Further, in some embodiments, the communication device 702 may beconfigured for receiving an administrator command from an administratordevice. Further, the generating of the at least one first virtual objectmodel may be based on the administrator command. Further, in someembodiments, the communication device 702 may be configured fortransmitting at least one second presentation data to at least onesecond presentation device (such as the second presentation device 720)associated with the second vehicle 714. Further, the at least one secondpresentation device may include a second receiver (such as the secondreceiver 722) configured for receiving the at least one secondpresentation data over the second communication channel. Further, the atleast one second presentation device may be configured for presentingthe at least one second presentation data. Further, in some embodiments,the processing device 704 may be configured for generating the at leastone second presentation data based on the at least one first sensordata. Further, in some embodiments, the storage device 706 may beconfigured for storing the at least one second presentation data.Further, in some embodiments, the storage device 706 may be configuredfor storing a first three-dimensional model corresponding to the firstvehicle 708. Further, the generating of the second presentation data maybe based on the first three-dimensional model. Further, in someembodiments, the at least one second presentation data may include atleast one second virtual object model corresponding to at least onesecond virtual object. Further, the generating of the at least onesecond virtual object model may be independent of the at least one firstsensor data. Further, the storage device 706 may be configured forstoring the at least one second virtual object model.

FIG. 8 is a flowchart of a method 800 of facilitating provisioning of avirtual experience, in accordance with some embodiments. Accordingly, at802, the method 800 may include receiving, using a communication device(such as the communication device 702), at least one first sensor datacorresponding to at least one first sensor (such as the at least firstsensor 710) associated with a first vehicle (such as the first vehicle708). Further, the at least one first sensor may be communicativelycoupled to a first transmitter (such as the first transmitter 712)configured for transmitting the at least one first sensor data over afirst communication channel.

Further, at 804, the method 800 may include receiving, using thecommunication device, at least one second sensor data corresponding toat least one second sensor (such as the at least one second sensor 716)associated with a second vehicle (such as the second vehicle 714).Further, the at least one second sensor may include a second locationsensor configured to detect a second location associated with the secondvehicle. Further, the at least one second sensor may be communicativelycoupled to a second transmitter (such as the second transmitter 718)configured for transmitting the at least one second sensor data over asecond communication channel. Further, in some embodiments, the at leastone second sensor may include a second user sensor configured forsensing a second user variable associated with a second user of thesecond vehicle. Further, the second user variable may include a seconduser location and a second user orientation.

Further, at 806, the method 800 may include transmitting, using thecommunication device, at least one second presentation data to at leastone second presentation device (such as the at least one secondpresentation device 720) associated with the second vehicle. Further,the at least one second presentation data may include at least onesecond virtual object model corresponding to at least one second virtualobject. Further, in some embodiments, the at least one second virtualobject may include one or more of a navigational marker (such as anavigational marker 1308, and/or a signboard 1504 as shown in FIG. 15)and an air-corridor (such as a skyway 1306 as shown in FIG. 13).Further, the at least one second presentation device may include asecond receiver (such as the second receiver 722) configured forreceiving the at least one second presentation data over the secondcommunication channel. Further, the at least one second presentationdevice may be configured for presenting the at least one secondpresentation data. Further, in some embodiments, the at least one secondpresentation device may include a second head mount display. Further,the second head mount display may include a second user location sensorof the at least one second sensor configured for sensing the second userlocation and a second user orientation sensor of the at least one secondsensor configured for sensing the second user orientation. Further, thesecond head mount display may include a second see-through displaydevice.

Further, at 808, the method 800 may include generating, using aprocessing device (such as the processing device 704), the at least onesecond presentation data based on the at least one first sensor data andthe at least one second sensor data. Further, the generating of the atleast one second virtual object model may be independent of the at leastone first sensor data.

Further, at 810, the method 800 may include storing, using a storagedevice (such as the storage device 706), the at least one secondpresentation data.

Further, in some embodiments, the method 800 may include retrieving,using the storage device, the at least one second virtual object modelbased on the second location associated with the second vehicle.

Further, in some embodiments, the method 800 may include determining,using the processing device, a second airspace class (with reference toFIG. 14) associated with the second vehicle based on the second locationincluding a second altitude associated with the second vehicle. Further,the generating of the at least one second virtual object model may bebased on the second airspace class.

Further, in some embodiments, the method 800 may include storing, usingthe storage device, a first three-dimensional model corresponding to thefirst vehicle. Further, the generating of the second presentation datamay be based on the first three-dimensional model.

Further, in some embodiments, the method 800 may include receiving,using the communication device, an administrator command from anadministrator device. Further, the generating of the at least one secondvirtual object model may be based on the administrator command.

Further, in some exemplary embodiments, the method 800 may includereceiving, using a communication device (such as the communicationdevice 702), at least one second sensor data corresponding to at leastone second sensor (such as the at least second sensor 716) associatedwith a second vehicle (such as the second vehicle 714). Further, the atleast one second sensor may be communicatively coupled to a secondtransmitter (such as the second transmitter 718) configured fortransmitting the at least one second sensor data over a secondcommunication channel. Further, the method 800 may include receiving,using the communication device, at least one first sensor datacorresponding to at least one first sensor (such as the at least onefirst sensor 710) associated with a first vehicle (such as the firstvehicle 708). Further, the at least one first sensor may include a firstlocation sensor configured to detect a first location associated withthe first vehicle. Further, the at least one first sensor may becommunicatively coupled to a first transmitter (such as the firsttransmitter 712) configured for transmitting the at least one firstsensor data over a first communication channel. Further, in someembodiments, the at least one first sensor may include a first usersensor configured for sensing a first user variable associated with afirst user of the first vehicle. Further, the first user variable mayinclude a first user location and a first user orientation. Further, themethod 800 may include transmitting, using the communication device, atleast one first presentation data to at least one first presentationdevice associated with the first vehicle 708. Further, the at least onefirst presentation data may include at least one first virtual objectmodel corresponding to at least one first virtual object. Further, insome embodiments, the at least one first virtual object may include oneor more of a navigational marker (such as a navigational marker 1308,and/or a signboard 1504 as shown in FIG. 15) and an air-corridor (suchas a skyway 1306 as shown in FIG. 13). Further, the at least one firstpresentation device may include a first receiver configured forreceiving the at least one first presentation data over the firstcommunication channel. Further, the at least one first presentationdevice may be configured for presenting the at least one firstpresentation data. Further, in some embodiments, the at least one firstpresentation device may include a first head mount display. Further, thefirst head mount display may include a first user location sensor of theat least one first sensor configured for sensing the first user locationand a first user orientation sensor of the at least one first sensorconfigured for sensing the first user orientation. Further, the firsthead mount display may include a first see-through display device.Further, the method 800 may include generating, using a processingdevice (such as the processing device 704), the at least one firstpresentation data based on the at least one second sensor data and theat least one first sensor data. Further, the generating of the at leastone first virtual object model may be independent of the at least onesecond sensor data. Further, the method 800 may include storing, using astorage device (such as the storage device 706), the at least one firstpresentation data. Further, in some embodiments, the method 800 mayinclude retrieving, using the storage device, the at least one firstvirtual object model based on the first location associated with thefirst vehicle 708. Further, in some embodiments, the method 800 mayinclude determining, using the processing device, a first airspace class(with reference to FIG. 14) associated with the first vehicle 708 basedon the first location including a first altitude associated with thefirst vehicle. Further, the generating of the at least one first virtualobject model may be based on the first airspace class. Further, in someembodiments, the method 800 may include storing, using the storagedevice, a second three-dimensional model corresponding to the secondvehicle 714. Further, the generating of the first presentation data maybe based on the second three-dimensional model. Further, in someembodiments, the method 800 may include receiving, using thecommunication device, an administrator command from an administratordevice. Further, the generating of the at least one first virtual objectmodel may be based on the administrator command.

FIG. 9 is a flowchart of a method 900 to facilitate providing at leastone first presentation data. Accordingly, at 902, the method 900 mayinclude transmitting, using the communication device, at least one firstpresentation data to at least one first presentation device associatedwith the first vehicle. Further, the at least one first presentationdevice may include a first receiver configured for receiving the atleast one first presentation data over the first communication channel.Further, the at least one first presentation device may be configuredfor presenting the at least one first presentation data.

Further, at 904, the method 900 may include generating, using theprocessing device, the at least one first presentation data based on theat least one second sensor data.

Further, at 906, the method 900 may include storing, using the storagedevice, the at least one first presentation data.

Further, in some embodiments, the method 900 may include storing, usingthe storage device, a second three-dimensional model corresponding tothe second vehicle. Further, the generating of the first presentationdata may be based on the second three-dimensional model.

Further, in some embodiments, the at least one first presentation datamay include at least one first virtual object model corresponding to atleast one first virtual object.

Further, the generating of the at least one first virtual object modelmay be independent of the at least one second sensor data. Further, themethod 900 may include storing, using the storage device, the at leastone first virtual object model.

Further, in some exemplary embodiment, the method 900 may facilitateproviding at least one second presentation data. Accordingly, the method900 may include transmitting, using the communication device, at leastone second presentation data to at least one second presentation deviceassociated with the second vehicle. Further, the at least one secondpresentation device may include a second receiver configured forreceiving the at least one second presentation data over the secondcommunication channel. Further, the at least one second presentationdevice may be configured for presenting the at least one secondpresentation data. Further, the method 900 may include generating, usingthe processing device, the at least one second presentation data basedon the at least one first sensor data. Further, the method 900 mayinclude storing, using the storage device, the at least one secondpresentation data. Further, in some embodiments, the method 900 mayinclude storing, using the storage device, a first three-dimensionalmodel corresponding to the first vehicle. Further, the generating of thesecond presentation data may be based on the first three-dimensionalmodel. Further, in some embodiments, the at least one secondpresentation data may include at least one second virtual object modelcorresponding to at least one second virtual object. Further, thegenerating of the at least one second virtual object model may beindependent of the at least one first sensor data. Further, the method900 may include storing, using the storage device, the at least onesecond virtual object model.

FIG. 10 shows a method 1000 to allow real pilots in real aircraft usingaugmented and virtual reality to meet in a virtual airspace, inaccordance with some embodiments. Accordingly, at 1002, the method 1000may include creating the virtual airspace in an augmented and virtualreality environment. The virtual airspace may be a three-dimensionalspace in which one or more aircraft may meet.

Further, at 1004, the method 1000 may include a real pilot in a realaircraft joining the virtual airspace via their augmented and virtualreality equipment. The real aircraft may be flying in the real world.Accordingly, an image of the real aircraft may be included in thevirtual airspace. Therefore, this provides a live simulation involvingreal people operating real systems.

In some embodiments, the virtual airspace may include virtual aircraft,which may be flown by real people in simulated systems, on the ground.

In some embodiments, the virtual airspace may further includeconstructed aircraft (and/or targets). The constructed aircraft may begenerated and controlled using computer graphics and processing systems.

Further, at 1006, the method 1000 may include providing augmented andvirtual reality content to the real pilot via their augmented andvirtual reality equipment. In some embodiments, the method may includeproviding augmented and virtual reality content to the real people (onthe ground) flying virtual aircraft in the virtual airspace.

Further, at 1008, the method 1000 may include tracking the real pilotand the real aircraft. This may include tracking the position andorientation of the pilot's head within the cockpit of the aircraft usingthe one or more internal sensors. Further, this may include tracking theoperational state (e.g. location, speed, direction of travel, etc.) ofthe aircraft in the virtual airspace using the one or more externalsensors.

Moreover, at 1010, the method 1000 may include continuously updating theaugmented and virtual reality content shown to the real pilot flying thereal aircraft based on the tracking of the real pilot and the realaircraft.

In some embodiments, the augmented and virtual reality content shown tothe real pilot flying the real aircraft may be updated based on theoperational state (e.g. location, speed, direction of travel, etc.) ofthe virtual aircraft flown by the real people (on the ground) and theoperational state (e.g. location, speed, direction of travel, etc.) ofthe constructed aircraft.

In some embodiments, the method 1000 may include continuously updatingthe augmented and virtual reality content shown to the real pilot (onthe ground) flying the virtual aircraft based on the tracking the realpilot and the real aircraft, the operational state (e.g. location,speed, direction of travel, etc.) of the virtual aircraft flown by thereal people (on the ground) and the operational state (e.g. location,speed, direction of travel, etc.) of the constructed aircraft.

FIG. 11 shows the augmented and virtual reality content shown to a realpilot (such as pilot 1102) flying a real aircraft (such as aircraft1104), in accordance with an exemplary embodiment. The augmented andvirtual reality content may include one or more live aircraft 1106(representing real pilots flying real aircraft), one or more virtualaircraft 1108 (representing real people on the ground, flying virtualaircraft) and one or more constructed aircraft 1110 (representingaircraft generated and controlled using computer graphics and processingsystems). Accordingly, the pilot 1102 wearing an augmented and virtualreality display device may look out the cockpit window to see enemyaircraft (such as live aircraft 1106, virtual aircraft 1108, and/orconstructed aircraft 1110) in extremely high fidelity. Further, thepilot 1102 may then practice offensive/defensive air-to-air maneuversagainst the digital enemy while continuing to fly his own aircraft 1104.

Systems and methods according to the principles of the presentinventions relate to an augmented reality system adapted to provide avirtual common training environment to operators of vehicles that areseparated by a distance where they cannot see one another; however, thecommon training environment provides the separated operators to seecomputer generated representations of one another so they can maneuveras if they were within visual range of one another. This may be usefulwhen teaching separated operators to maneuver in a coordinated fashion(e.g. within a close formation of planes) when the operators cannototherwise see each other. In embodiments, two separate head-mountedsee-through optics, a first and a second, adapted to present digitalcontent viewable by a user and having a transparency that allows theuser to see though to the surrounding environment are provided to thevisually separated operators. A training simulation system may beadapted to present digital content to each of the first and secondoptics, wherein the digital content represents a vehicle operated by theother user. With this arrangement, each operator can ‘see’ the othervehicle as digital content in the see-through display such that they cancoordinate maneuvers. The digital representation of the other's vehiclemay be geospatially positioned based on the other vehicle's actualgeospatial position as represented in the common training environment.For example, the position of the digital representation of the othervehicle may be represent the other vehicle's actual geospatial locationand condition in its real airspace such that movements of the secondvehicle are duplicated by the representation of the second vehicle, butthe geospatial location of the representation in the virtual commontraining environment may be based on the training exercise such that theapparent distance from an operator to the representation is related andrelative.

Systems and methods according to the principles of the presentinventions relate to presenting of a coordinated training scenario totwo or more vehicles in separate airspaces where the two or morevehicles are not within visual range of one another. In embodiments, acommon virtual airspace is provided to the two or more vehicles, whereinthe common virtual airspace includes a computer generated training assetthat is viewable by an operator of each vehicle as content overlaying areal airspace surrounding each of the respective vehicles. It ispresented as augmented reality content. A system may identify ageospatial location for each of the two or more vehicles within thevirtual airspace, which may be based on the vehicle's actual geospatiallocations within their respective airspace and represented within thecommon virtual airspace. The system may position the computer generatedtraining asset at a geospatial location within the virtual airspacewithin a visual range of the two or more vehicles such that theperspective of the computer generated training asset is separately basedon the geospatial location for each of the two or more vehicles.

In embodiments, systems and methods may involve presenting a first pilotof a first vehicle, of the two or more vehicles, with computer generatedcontent representing a second vehicle, of the two of more vehicles,within a common virtual environment when the first pilot looks in thedirection of the second vehicle's geospatial location as mapped into thecommon virtual environment. This facilitates visual coordination betweenthe otherwise visually separated vehicles. For example, a real pilot ina real aircraft flying over Nevada may be able to ‘see’ a second planethat is actually flying over Virginia as an augmented realityrepresentation in close proximity to the real aircraft. The relativepositioning of the representation of the second aircraft to the realaircraft may be programmed based on the training scenario. The scenario,for example, may begin by geospatially locating the two visuallyseparated aircraft within 50 feet of one another in a common virtualairspace. Each pilot would be able to look and see the other's aircraftrepresented as augmented reality content at 50 feet away. Then thesimulation may track the actual geospatial positions and conditions ofboth aircraft to move the representations presented to the pilots basedon actual movements. If the either plane makes a real maneuver thataffects the relative position of the two aircraft in the virtualenvironment, the result will be shown by changing the position of therepresentation of the other aircraft. If the second aircraft puts on itsafterburners and the first does not, the pilot of the first aircraftwill see the second aircraft pull away in the virtual airspace as thesecond aircraft flies faster in its real airspace.

In embodiments, the presenting of the computer generated contentrepresenting the second vehicle is further based on an unobstructed lineof sight between the first pilot and the location computer contentrepresenting the second vehicle in the virtual environment. The apparentrelative position between the first vehicle and the computer generatedcontent representing the second vehicle may be based on the actualmovements of the first vehicle and a second vehicle. The computergenerated training asset and the computer generated content representingthe second vehicle may move separately within the virtual environment,the second vehicle representation movements may be based on the actualmovement of the second vehicle and the training asset movements may bebased on a computer generated path (e.g. predetermined path) intended tointeract with at least one of the two or more vehicles.

In embodiments, the geospatial boundaries of the virtual airspace areset such that each of the two or more vehicles operate withinrespectively clear airspace. In embodiments, the common virtual airspacerepresents one of the two or more vehicle's actual airspace. Inembodiments, the common virtual airspace represents an airspace in anenemy environment.

Systems and methods according to the principles of the presentinventions relate to providing a common virtual environment provided toboth air and ground assets. A combat training augmented realitysimulation system may be provided to real aircraft and real groundvehicle. The simulation may involve including virtual air and groundassets. In embodiments, at least two separate head-mounted see-throughoptics, a first and a second, are provided and adapted to presentdigital content viewable by a user and having a transparency that allowsthe user to see though to the surrounding environment. An aircraftmounted tracking systems mounted in the real aircraft may be used totrack the geo-spatial position and condition of the aircraft and thedirection in which a pilot is apparently looking. A ground vehicletracking system may also be mounted in the real ground vehicle so theground tracking system tracks the geo-spatial position of the realground vehicle and the direction in which a driver is apparentlylooking. A training simulation system adapted to generate 3D virtualcontent for presentation on the respective separate optics, wherein thevirtual content provides the pilot and the driver with a differentperspective view of the same 3D virtual object.

FIG. 12 shows two real aircraft (such as aircraft 1202, and aircraft1204) in a virtual airspace 1206, in accordance with an exemplaryembodiment. The two real aircraft (such as aircraft 1202, and aircraft1204) may be flown by two real pilots (a pilot A and a pilot B).Further, both the pilots may be capable of using the disclosed system(ATARI) to view the augmented and virtual reality content. Further, thepilot A may be able to see the pilot B via their augmented and virtualreality equipment. Further, the pilot A may be able to see one or morevirtual aircraft (not shown in FIG. 12) which may be enemy aircraft orfriendly aircraft.

In some embodiments, the pilot A and the pilot B may be enemies and mayengage in combat against each other.

In some embodiments, the pilot A and the pilot B may be friendly and maycooperate in combat against enemy aircraft. High-speed communicationbetween the two aircraft may be employed to allow for effectivecooperation.

In some embodiments, the two aircraft 1202-1204 may not fly together inthe real world. As shown in FIG. 12, one aircraft (such as aircraft1202) may take off in the USA and the other aircraft (such as aircraft1204) may take off in the UK. Therefore, the two aircraft 1202-1204 flyphysically in the air in different geographical location, but they mayshare the same virtual airspace (6D airspace) provided by the disclosedsystem (ATARI).

Accordingly, the pilot A may fight against the pilot B in the commonvirtual airspace 1206. Therefore, each pilot may see other pilot'svirtual image in their augmented and virtual reality equipment.

Further, the pilot A and the pilot B may fight together against enemies.Again, both pilots may see each other's virtual images. However, in thiscase, they may collaborate, and not fight against each other.

FIG. 13 shows an augmented reality view 1300 shown to a real pilot (suchas pilot 1302), in accordance with an exemplary embodiment. Further, theaugmented reality view 1300 may be generated and displayed over avirtual reality display. For example, the virtual reality display mayinclude a head-mounted display (HMD), eyeglasses, Head-Up Display (HUD),smart contact lenses, a virtual retinal display, an eye tap, a PrimaryFlight Display (PFD) and a cockpit glass etc. Further, the augmentedreality view 1300 may assist a pilot 1302 in flying a civilian aircraft1304.

As shown in FIG. 13, the augmented reality view 1300 includes a roaddrawn in the sky (such as a skyway 1306) indicating a path that thecivilian aircraft 1304 may take in order to land at an airport. Further,the augmented reality view 1300 may include a navigation marker 1308indicating to the pilot 1302 that the civilian aircraft 1304 should takea left turn. The navigation marker 1308 may assist the pilot 1302 innavigating towards a landing strip to land the civilian aircraft 1304.

Therefore, the augmented reality view 1300 may provide pilots with asimilar view as seen by public transport drivers (e.g. taxi or bus) onthe ground. The pilots (such as the pilot 1302) may see roads (such asthe skyway 1306) that the pilot 1302 need to drive on. Further, thepilot 1302, in an instance, may see signs just like a taxi driver whomay just look out of a window and see road signs.

Further, the augmented reality view 1300 may include (but not limitedto) one or more of skyways (such the skyway 1306), navigation markers(such as the navigation marker 1308), virtual tunnels, weatherinformation, an air corridor, speed, signboards for precautions,airspace class, one or more parameters shown on a conventionalhorizontal situation indicator (HSI) etc. The skyways may indicate apath that an aircraft (such as the civilian aircraft 1304) should take.The skyways may appear similar to roads on the ground. The navigationmarkers may be similar to regulatory road signs used on the roads on theground. Further, the navigation markers may instruct pilots (such as thepilot 1302) on what they must or should do (or not do) under a given setof circumstances. Further, the navigation markers may be used toreinforce air-traffic laws, regulations or requirements which applyeither at all times or at specified times or places upon a flight path.For example, the navigation markers may include one or more of a leftcurve ahead sign, a right curve ahead sign, a keep left sign, and a keepto right sign. Further, the virtual tunnels may appear similar totunnels on roads on the ground. The pilot 1302 may be required to flythe aircraft through the virtual tunnel. Further, the weatherinformation may include real-time weather data that affects flyingconditions. For example, the weather information may include informationrelated to one or more of wind speed, gust, and direction; variable winddirection; visibility, and variable visibility; temperature;precipitation; and cloud cover. Further, the air corridor may indicatean air route along which the aircraft is allowed to fly, especially whenthe aircraft is over a foreign country. Further, the augmented realityview 1300 may include speed information. The speed information mayinclude one or more of a current speed, a ground speed, and arecommended speed. The signboards for precautions may be related towarnings shown to the pilot 1302. The one or more parameters shown on aconventional horizontal situation indicator (HSI) include NAV warningflag, lubber line, compass warning flag, course select pointer, TO/FROMindicator, glideslope deviation scale, heading select knob, compasscard, course deviation scale, course select knob, course deviation bar(CDI), symbolic aircraft, dual glideslope pointers, and heading selectbug.

Further, in some embodiments, information such as altitude, attitude,airspeed, the rate of climb, heading, autopilot and auto-throttleengagement status, flight director modes and approach status etc. thatmay be displayed on a conventional primary flight display may also bedisplayed in the augmented reality view 1300.

Further, in some embodiments, the augmented reality view 1300 mayinclude a one or more of other vehicles (such as another airplane 1310).Further, the one or more other vehicles, in an instance, may include oneor more live vehicles (such as representing real pilots flying realaircraft), one or more virtual vehicles (such as representing realpeople on the ground, flying virtual aircraft), and one or moreconstructed vehicles (such as representing aircraft generated andcontrolled using computer graphics and processing systems).

Further, the augmented reality view 1300 may include an airspace. FIG.14 is a chart related to the United States airspace system'sclassification scheme. Specifically, FIG. 14 illustrates variousparameters related to one or more classes defined in the United Statesairspace system's classification scheme. The classification scheme isintended to maximize pilot flexibility within acceptable levels of riskappropriate to the type of operation and traffic density within thatclass of airspace—in particular, to provide separation and activecontrol in areas of dense or high-speed flight operations. The AlbertRoper (1919-10-13 The Paris Convention) implementation of InternationalCivil Aviation Organization (ICAO) airspace classes defines classes Athrough G (with the exception of class F which is not used in the UnitedStates).

For an instance, a computing device (such as the computing device 1600)may analyze one or more parameters such as altitude, Visual Flight Rules(VFR), Instrument Flight Rules (IFR), VFR cloud clearance, and VFRminimum visibility etc. to determine an applicable airspace class.Further, the determined airspace class may be displayed on the virtualreality display. Further, the applicable airspace class may bedetermined using a location tracker such as a GPS and may be displayedas a notification on the virtual reality display.

Further, a special use airspace class may be determined. The special useairspace class may include alert areas, warning areas, restricted areas,prohibited airspace, military operation area, national security area,controlled firing areas etc. For an instance, if an aircraft (such asthe civilian aircraft 1304) enters a prohibited area by mistake, then anotification may be displayed in the augmented reality view 1300.Accordingly, the pilot 1302 may reroute the aircraft towards a permittedairspace.

Further, the augmented reality view 1300 may include one or more liveaircraft (representing real pilots flying real aircraft), one or morevirtual aircraft (representing real people on the ground, flying virtualaircraft) and one or more constructed aircraft (representing aircraftgenerated and controlled using computer graphics and processingsystems). Further, the augmented reality view 1300 shown to a pilot(such as the pilot 1302) in a first aircraft (such as the civilianaircraft 1304) may be modified based on sensor data received fromanother aircraft (such as another airplane 1310). The sensor data mayinclude data received from one or more internal sensors to track andlocalize the pilot's head within the cockpit of the aircraft. Further,the sensor data may include data received from one or more externalsensors to track the position and orientation of the aircraft. Further,the data received from the one or more internal sensors and the one ormore external sensors may be combined to provide a highly usableaugmented reality solution in a fast-moving environment.

FIG. 15 shows an augmented reality view 1500 shown to a real pilot whilea civilian aircraft 1502 is taxiing at an airport, in accordance with anexemplary embodiment. The augmented reality view 1500 may include one ormore navigational markers (such as the navigation marker 1308) andsignboards (such as a signboard 1504) that assist a pilot to taxi thecivilian aircraft 1502 at the airport. The navigational markers mayindicate the direction of movement. The signboards may indicate thespeed limits.

The augmented reality view 1500 may help the pilot to taxi the civilianaircraft 1502 towards a parking location after landing. Further,augmented reality view 1500 may help the pilot to taxi the civilianaircraft 1502 towards a runway for taking-off. Therefore, a ground crewmay no longer be required to instruct the pilot while taxiing thecivilian aircraft 1502 at the airport.

Further, the augmented reality view 1500 may include one or more liveaircraft (such as a live aircraft 1506) at the airport (representingreal pilots in real aircraft), one or more virtual aircraft at theairport (representing real people on the ground, controlling a virtualaircraft) and one or more constructed aircraft at the airport(representing aircraft generated and controlled using computer graphicsand processing systems). Further, the augmented reality view 1500 shownto a pilot in a first aircraft may be modified based on sensor datareceived from another aircraft. The sensor data may include datareceived from one or more internal sensors to track and localize thepilot's head within the cockpit of the aircraft. Further, the sensordata may include data received from one or more external sensors totrack the position and orientation of the aircraft. Further, the datareceived from the one or more internal sensors and the one or moreexternal sensors may be combined to provide a highly usable augmentedreality solution in a fast-moving environment.

With reference to FIG. 16, a system consistent with an embodiment of thedisclosure may include a computing device or cloud service, such ascomputing device 1600. In a basic configuration, computing device 1600may include at least one processing unit 1602 and a system memory 1604.Depending on the configuration and type of computing device, systemmemory 1604 may comprise, but is not limited to, volatile (e.g.random-access memory (RAM)), non-volatile (e.g. read-only memory (ROM)),flash memory, or any combination. System memory 1604 may includeoperating system 1605, one or more programming modules 1606, and mayinclude a program data 1607. Operating system 1605, for example, may besuitable for controlling computing device 1600's operation. In oneembodiment, programming modules 1606 may include virtualization module,image-processing module, machine learning module and/or tracking module.Furthermore, embodiments of the disclosure may be practiced inconjunction with a graphics library, other operating systems, or anyother application program and is not limited to any particularapplication or system. This basic configuration is illustrated in FIG.16 by those components within a dashed line 1608.

Computing device 1600 may have additional features or functionality. Forexample, computing device 1600 may also include additional data storagedevices (removable and/or non-removable) such as, for example, magneticdisks, optical disks, or tape. Such additional storage is illustrated inFIG. 16 by a removable storage 1609 and a non-removable storage 1610.Computer storage media may include volatile and nonvolatile, removableand non-removable media implemented in any method or technology forstorage of information, such as computer-readable instructions, datastructures, program modules, or other data. System memory 1604,removable storage 1609, and non-removable storage 1610 are all computerstorage media examples (i.e., memory storage.) Computer storage mediamay include, but is not limited to, RAM, ROM, electrically erasableread-only memory (EEPROM), flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to storeinformation and which can be accessed by computing device 1600. Any suchcomputer storage media may be part of device 1600. Computing device 1600may also have input device(s) 1612 such as a keyboard, a mouse, a pen, asound input device, a touch input device, a location sensor, a camera, abiometric sensor, etc. Output device(s) 1614 such as a display,speakers, a printer, etc. may also be included. The aforementioneddevices are examples and others may be used.

Computing device 1600 may also contain a communication connection 1616that may allow device 1600 to communicate with other computing devices1618, such as over a network in a distributed computing environment, forexample, an intranet or the Internet. Communication connection 1616 isone example of communication media. Communication media may typically beembodied by computer readable instructions, data structures, programmodules, or other data in a modulated data signal, such as a carrierwave or other transport mechanism, and includes any information deliverymedia. The term “modulated data signal” may describe a signal that hasone or more characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media may include wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, radiofrequency (RF), infrared, and other wireless media. The term computerreadable media as used herein may include both storage media andcommunication media.

As stated above, a number of program modules and data files may bestored in system memory 1604, including operating system 1605. Whileexecuting on processing unit 1602, programming modules 1606 (e.g.,application 1620 such as a media player) may perform processesincluding, for example, one or more stages of methods, algorithms,systems, applications, servers, databases as described above. Theaforementioned process is an example, and processing unit 1602 mayperform other processes. Other programming modules that may be used inaccordance with embodiments of the present disclosure may include soundencoding/decoding applications, machine learning application, acousticclassifiers etc.

There are a number of ways to track aircraft and other vehicles. You canwatch a vehicle move within your own local environment or in alonger-range environment with the aid of magnification. You can trackthings with radar when they are within visual sight or beyond visualsight. Radar systems may be local (e.g. within your vehicle) or remote(e.g. within another vehicle, on the ground separate, in the air, inspace, etc.). Satellite imagery and other satellite sensor systems cantrack things. And other systems can be adapted to track vehicles, peopleand objects that are in a person's near environment or far from theperson.

Military training exercises typically use a number of trackingtechnologies. In a situation where there are a number of real vehiclesacting as enemies each vehicle may be equipped with trackingtechnologies to track the opponent. As the two vehicles (e.g. fighterjets) come within visual range of one another the pilot's will alsolikely track the opposition visually. This may be a practice dogfighting scenario. Each pilot would be able to track the opponentthrough all available tracking system, including her own vision. Theinventors discovered that this places a significant burden on trainingexercises. There is a real risk of loss of life and loss of assets.There is a massive expense in operating assets in these trainingscenarios. The situations are limited to those of the skills of theindividual pilots and other vehicle operators and restricted by theexercise rules. The rules may limit the participants from performingextreme maneuvers, for example, to minimize the risk to life and assets.

To incorporate more extreme conditions, preserve life and assets, savemoney, and train many more vehicle operators, ground based simulatorsare used for visual contact situations. Some ground-based simulators arevery complex and can provide a near real feeling of being in a vehiclein certain situations, but they are limited because they are not in realvehicles and the simulations of a hostile cockpit environment, forexample, is very difficult to accurately simulate. To simulate a turn at5G's a ground-based simulator may move the operator and place pressureon the operator, but it is not like pulling 5G's in a real airplane. Inthe real jet fighter, the situation could go well above 5G's, riskingthe pilot to passing out and/or reduced cognitive ability, which greatlyimpact the pilot's ability to think and react.

Systems and methods according to the present disclosure use vehicletracking and operator vision direction tracking as described herein toproperly manage the presentation of virtual content within a realvehicle to simulate and augment the environment outside of the vehicle.Such systems and methods may further integrate non-visual trackingtechnologies. For example, a pilot in a fighter jet may have a radarsystem, whether local radar or other radar, to track assets out ofvisual range (e.g. approximately 10 miles away in a clear sky, much lessin a cloudy sky). In a training simulation, the radar system may bepresenting information not about another real vehicle but, rather, avirtual asset being controlled by a simulation system. The pilot may usethe simulated radar information and maneuver the plane in a trainingexercise. This provides the pilot with the real environment of thecockpit while reacting in a simulation, which can provide a very goodsimulation with respect to non-visual tracking of assets.

In embodiments, the in-vehicle augmented reality system (e.g. HMD)simulation content is coordinated with non-visual tracking technologies.For example, a plane's radar system may indicate that an enemy plane isapproaching at a fast rate and from a certain direction. When the enemyis more than 10 miles away, for example, the only indication the pilotmay have of the enemy is the radar information. As the enemy approachesand comes within the pilot's visual range, the augmented reality systemand simulation system may coordinate to ‘hand off’ the non-visualrepresentation of the enemy to an augmented reality view of the enemy.The non-visual portion of the simulation may continue to provideinformation to the pilot is engaged in the augmented reality view. Thetransition from non-visual data to visual presentation of content isintended to be smooth and life like. For example, the size of the visualrepresentation would be very small at the initial transition and itwould follow the path indicated by the non-visual system. The trackingin the non-visual and virtual visual may coordinate throughout thevisual range for the pilot such that she does not have any inconsistentinput.

The non-visual and visual coordination may be generated, monitored,corrected, etc. in real time by a computer system. The computer systemmay be a central system that controls or influences the virtualenvironment and content presented to a vehicle operator or passenger.The computer system may be arranged to coordinate the simulatednon-visual content that is presented to the operator of a vehicle withvisual content that is presented to the operator such that the operatorperceives the two information feeds as coordinated. The visual contentwould be displayed at a geospatial position and condition (e.g.perspective geometry) that is consistent with the simulated non-visualcontent's apparent location and condition (e.g. perspective geometry,location, speed, etc.).

In embodiments, coordination between non-visual information concerninganother vehicle or object in a vehicle's environment may berepresentative of a real vehicle or object (e.g. radar tracking andpresentation of another airplane or missel more than 10 miles away).Once the real object enters the vehicle operator's visual range, virtualvisual content may be presented to the operator in his see-throughoptical system as augmented reality content. The virtual content mayprovide the operator with information pertaining to the other vehicle orobject and/or cues to guide the operator with respect to the othervehicle or object.

A system and method according to the principles of the present inventionmay be an augmented reality system for the training of vehicle operatorsinvolving visual and non-visual information. It may involve ahead-mounted see-through optic (e.g. HMD) adapted to present digitalcontent viewable by a user and having a transparency that allows theuser to see though to the surrounding environment. It may also involve anon-visual tracking system adapted to identify and track objects in asurrounding environment that cannot be seen visually. A trainingsimulation system may be adapted to present a virtual training object ona display on the non-visual tracking system and virtual visual contentin the see-through optic. Both the displays, HMD and non-visual display,may be representing a location and movement of the same training objectand the presentation maybe coordinated such that both displays indicatethe training object is in the same position.

A system and method according to the principles of the present inventionmay involve tracking and coordinating visual information and non-visualinformation relating to a virtual object in a training simulation. Thismay involve providing a non-visual object tracking system in a vehicleand providing an augmented reality see-through computer display adaptedto present virtual content representing an object to an operator of thevehicle. A training simulation system may generate a geospatial locationand path of movement of the virtual object at a geospatial locationoutside of a visual range of the operator. The geospatial location andpath of movement of the virtual object may be displayed on thenon-visual object tracking system while the object maintains a distancefrom the vehicle that is outside of the operator's visual range. Thesystem may present a representation of the virtual object in theoperator's see-through computer display when the location of the objectenters the operator's visual range. The representation may be presentedat a position within a field of view of the see-through computer displaythat is consistent with the position of the object as presented on thedisplay of the non-visual object tracking system.

In embodiments, the non-visual tracking system may be a radar trackingsystem. In embodiments, the virtual object may be an enemy asset.

In embodiments, the step of presenting the virtual object on the displayof the non-visual object tracking system may be part of a simulatedtraining exercise where the computer simulation system generates thevirtual object and determines the virtual objects path of movement. Thesystem may coordinate a substantially simultaneous presentation of thevisual representation of the virtual object and the non-visualrepresentation of the virtual object. In embodiments, the step ofcoordination involves alignment of the geospatial location and directionof movement consistent in the see-through computer display and thenon-visual display.

Systems and methods according to the principles of the presentinventions involve replaying live simulated training sessions. A livesimulated training session may involve a real vehicle operating in areal environment (as described herein) and an operator's and orvehicle's reactions to a virtual object presented as if within visualrange. The content may be presented as augmented reality content to theoperator. In embodiments, such a system and method may involve savingin-flight data from an aircraft during a simulated training exercise,wherein the in-flight data includes geospatial locations of theaircraft, positional attitudes of the aircraft, and head positions of apilot operating the aircraft. It may further involve saving simulationdata relating to a simulated virtual object presented to the pilot asaugmented reality content in-flight, wherein the virtual object wasprogrammed to interact with the aircraft during the simulated trainingexercise. With the relevant data from the augmented reality simulatedtraining saved, the system may represent the in-flight data from theaircraft and the simulation data relating to the simulated virtualobject as a replay of the simulated training exercise. The replay may bereviewed in real time, slow motion, fast forward, etc. to understand orteach lessons based on the training session.

Systems and methods described herein can be used to create a virtualenvironment presented to an operator or a passenger in a real operatingvehicle. As disclosed herein, the virtual environment may be used for avariety of uses (e.g. combat training, operational training, navigationguidance, visual cues for situations occurring during operation). Thevirtual environment maybe generated and/or controlled in-part orentirely by an artificial intelligence system, machine learning system,deep learning system, etc. (AI). For example, simulations of situationsmay have been completed in on-ground simulation systems or live vehiclesystems and the operator performance in those simulations may affect thecontrol of virtual objects in the in-vehicle virtual environment. Forexample, if a pilot usually turns the plane, his eyes, or his head in acertain direction in a certain situation, the AI may control orinfluence the content to try to cause the pilot to turn his head orchange something to cause some response.

As has been described herein, systems and methods disclosed herein maybe used for training, gaming, cueing an operator or passenger duringtraining or in a non-simulated, live, situation, coordinatinginformation amongst various participants (e.g. keeping vehicles in aformation), etc. While many of the embodiments herein describe trainingsimulations it should be understood that the principles of the presentinventions may relate to cueing an operator in a live situation.

With the introduction of systems and methods of visual cueing andtraining, the inventors discovered that the systems and methods may beused to generate and analyze an entirely new data type relating tofeedback from the in-vehicle situations. For example, a pilot may be ina real jet fighter flying at the speed of sound using an augmentedreality system as described herein. The system may present visualcontent to the pilot in augmented reality causing a response by thepilot and the response, biomarkers from the pilot or passenger, resultsof the response, etc. may be recorded, stored and analyzed by a computersystem. The data may be used to train an AI system, form trend analysisfor a group of people in similar situations, form personal trendanalysis for individual pilots or passengers, identify reaction timesfor the pilot and the vehicle, form a trend analysis of the vehicle'sconditions through real maneuvers, etc. Models, trends, situationalreactions from the analysis can be used to train groups of vehicleoperators, individual vehicle operators, provide trainers feedback,modify virtual environments and content in simulations and/or thecontrol of the content in simulations, modify cues presented tooperators in non-simulated, live, situations, etc. Generally, consistentwith embodiments of the disclosure, program modules may includeroutines, programs, components, data structures, and other types ofstructures that may perform particular tasks or that may implementparticular abstract data types. Moreover, embodiments of the disclosuremay be practiced with other computer system configurations, includinghand-held devices, general purpose graphics processor-based systems,multiprocessor systems, microprocessor-based or programmable consumerelectronics, application specific integrated circuit-based electronics,minicomputers, mainframe computers, and the like. Embodiments of thedisclosure may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

Furthermore, embodiments of the disclosure may be practiced in anelectrical circuit comprising discrete electronic elements, packaged orintegrated electronic chips containing logic gates, a circuit utilizinga microprocessor, or on a single chip containing electronic elements ormicroprocessors. Embodiments of the disclosure may also be practicedusing other technologies capable of performing logical operations suchas, for example, AND, OR, and NOT, including but not limited tomechanical, optical, fluidic, and quantum technologies. In addition,embodiments of the disclosure may be practiced within a general-purposecomputer or in any other circuits or systems.

Embodiments of the disclosure, for example, may be implemented as acomputer process (method), a computing system, or as an article ofmanufacture, such as a computer program product or computer readablemedia. The computer program product may be a computer storage mediareadable by a computer system and encoding a computer program ofinstructions for executing a computer process. The computer programproduct may also be a propagated signal on a carrier readable by acomputing system and encoding a computer program of instructions forexecuting a computer process. Accordingly, the present disclosure may beembodied in hardware and/or in software (including firmware, residentsoftware, micro-code, etc.). In other words, embodiments of the presentdisclosure may take the form of a computer program product on acomputer-usable or computer-readable storage medium havingcomputer-usable or computer-readable program code embodied in the mediumfor use by or in connection with an instruction execution system. Acomputer-usable or computer-readable medium may be any medium that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, apparatus, ordevice.

The computer-usable or computer-readable medium may be, for example butnot limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, device, or propagationmedium. More specific computer-readable medium examples (anon-exhaustive list), the computer-readable medium may include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a random-access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, and a portable compact disc read-only memory(CD-ROM). Note that the computer-usable or computer-readable mediumcould even be paper or another suitable medium upon which the program isprinted, as the program can be electronically captured, via, forinstance, optical scanning of the paper or other medium, then compiled,interpreted, or otherwise processed in a suitable manner, if necessary,and then stored in a computer memory.

Embodiments of the present disclosure, for example, are described abovewith reference to block diagrams and/or operational illustrations ofmethods, systems, and computer program products according to embodimentsof the disclosure. The functions/acts noted in the blocks may occur outof the order as shown in any flowchart. For example, two blocks shown insuccession may in fact be executed substantially concurrently or theblocks may sometimes be executed in the reverse order, depending uponthe functionality/acts involved.

While certain embodiments of the disclosure have been described, otherembodiments may exist. Furthermore, although embodiments of the presentdisclosure have been described as being associated with data stored inmemory and other storage mediums, data can also be stored on or readfrom other types of computer-readable media, such as secondary storagedevices, like hard disks, solid state storage (e.g., USB drive), or aCD-ROM, a carrier wave from the Internet, or other forms of RAM or ROM.Further, the disclosed methods' stages may be modified in any manner,including by reordering stages and/or inserting or deleting stages,without departing from the disclosure.

Although the invention has been explained in relation to its preferredembodiment, it is to be understood that many other possiblemodifications and variations can be made without departing from thespirit and scope of the invention.

1-12. (canceled)
 13. A method comprising: presenting a generated firstvirtual environment to a pilot operating an aircraft; detecting andsaving data indicative of one or more reactions by the pilot in responseto the first virtual environment; analyzing the saved data to determinea correlation between an identified attribute of the virtual environmentand at least one of the one or more reactions; and presenting a secondgenerated virtual environment comprising the identified attribute to thepilot to induce the at least one of the one or more reactions by thepilot.
 14. The method of claim 13, wherein the analyzing may beperformed by a system selected from the group consisting of anartificial intelligence system, a machine learning system and a deeplearning system.